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ABSTRACT 


The  Janus  simulation  model  was  initially  designed  to  operate  in  a  stand-alone 
mode.  There  is  an  ongoing  research  project  to  link  Janus  to  other  constructive 
simulations  and  virtual  simulators.  The  present  standard  used  to  connect  different  models 
is  Distributed  Interactive  Simulation  (DIS).  Janus  can  operate  in  a  DIS  environment 
using  a  cell  adapter  unit  called  the  World  Modeler.  The  combination  of  Janus  and  the 
World  Modeler  is  known  as  JLink.  A  goal  of  the  JLink  system  is  to  replicate  the 
analytical  and  training  fidelity  of  stand-alone  Janus  in  a  distributed  exercise.  The 
purpose  of  this  thesis  is  to  assess  the  current  state  of  JLink  development. 

The  experiment  simulated  three  scenarios:  armored,  armored  coalition,  and  light 
infantry  battalions  attacking  against  a  defending  company.  All  scenarios  were  executed 
in  two  contrasting  environments.  The  simulation  included  the  recently  developed  JLink 
features  Family  of  Scatterable  Mine  (FASCAM)  and  chemical  artillery. 

The  thesis  used  five  Measures  of  Performance  to  base  the  assessment:  1) 
FASCAM  kills,  2)  Chemical  Artillery  Kills,  3)  Detection  Ranges,  4)  Kill  Ranges,  and  5) 
Loss  Exchange  Ratio.  The  statistical  tests  used  for  analysis  were  the  Analysis  of 
Variance  (ANOVA)  test,  two-sample  r-test,  and  Wilcoxon  test. 

The  results  of  the  analysis  show  that  JLink  requires  adjustments  to  artillery 
delivery  methods  in  order  to  correct  chemical  artillery  discrepancies  and  detection  range 
issues.  In  general,  JLink  accurately  portrays  coalition  warfare  and  satisfactorily  replicates 
armored  and  infantry  scenarios  in  contrasting  environments. 
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EXECUTIVE  SUMMARY 


The  Army  relies  heavily  on  computer  simulation  as  a  training  an  analytical  tool. 
One  simulation  the  Army  uses  is  Janus,  a  high  resolution  combat  simulation  model 
design  to  operate  in  a  stand-alone  mode.  There  is  an  ongoing  research  effort  to  link  Janus 
with  other  constructive  simulations  or  virtual  simulators.  The  present  standard  for  linking 
different  models  is  Distributed  Interactive  Simulation  (DIS).  Janus  can  operate  in  a  DIS 
environment  using  a  cell  adapter  unit  called  the  World  Modeler.  The  combination  of 
Janus  and  the  World  Modeler  is  known  as  JLink.  A  goal  of  JLink  is  to  replicate  the 
training  and  analytic  fidelity  of  stand-alone  Janus  in  a  distributed  (JLink)  exercise.  The 
level  of  fidelity  may  be  gauged  by  comparing  the  results  of  a  scenario  run  in  stand-alone 
Janus  with  the  results  from  the  same  scenario  run  in  JLink.  The  purpose  of  this  thesis  is 
to  assess  the  current  state  of  JLink  development  by  measuring  the  amount  of  distortion 
between  stand-alone  Janus  and  JLink. 

The  experiment  tested  specific  areas  of  JLink  which  had  yet  been  assessed. 
Previously,  only  rudimentary  scenarios  had  been  tested  in  a  single  environment.  This 
thesis  tested  scenarios  using  different  types  of  combat  units  in  two  different 
environments.  The  thesis  also  tested  JLink’ s  ability  to  portray  coalition  warfare.  Finally, 
the  thesis  assessed  recent  software  inputs  to  JLink  which  simulated  specific  combat 
functions,  in  particular  Family  of  Scatterable  Mines  (FASCAM)  and  chemical  artillery. 


xi 


The  experimental  design  used  to  assess  these  aspects  of  JLink  included  armored 
and  light  infantry  units.  The  first  scenario  involved  a  Soviet-equipped  armored  battalion 
attacking  against  a  defending  US-equipped  armored  company.  The  second  scenario  was 
identical  to  the  first  except  that  the  attacking  battalion  was  divided  into  two  separate 
forces,  both  attacking  the  armored  company.  This  scenario  was  used  to  assess  JLink’ s 
ability  to  replicate  a  coalition  battle.  The  last  scenario  involved  a  Soviet-equipped  light 
battalion  attacking  against  a  US-equipped  light  infantry  company.  All  three  scenarios 
were  executed  in  two  different  environments.  The  two  environments  selected  were  Fort 
Hunter  Liggett,  California  (HL)  and  Southwest  Asia  (SWA).  These  two  environments 
offered  contrasting  terrain,  the  first  being  wooded  with  rolling  hills  and  the  second  being 
flat  and  open.  Using  different  types  of  units  in  contrasting  terrain  permitted  assessment 
of  JLink’s  ability  to  portray  varied  scenarios.  To  test  the  specific  combat  functions  of 
FASCAM  and  chemical  kills,  the  defending  company  used  these  weapons  in  every 
scenario. 

The  Measures  of  Performance  (MOPs)  selected  for  analysis  were  FASCAM  kills, 
chemical  kills,  detection  range,  kill  range,  rounds  fired,  and  Loss  Exchange  Ratio  (LER). 
These  MOPs  allowed  for  analysis  of  the  basic  procedures  of  an  engagement.  First,  one 
entity  must  detect  another  entity.  Once  detected,  the  entity  may  elect  to  shoot  at  the 
target.  Analyzing  detection  range,  kill  range,  and  rounds  fired  offers  a  quantitative 
method  of  assessing  these  basic  engagement  procedures.  The  LER  is  a  function  of  the 


other  MOPs  and  measures  the  overall  outcome  of  the  battle.  The  MOPs  FASCAM  kills 


and  chemical  kills  were  used  to  assess  JLink’s  ability  to  simulate  those  functions. 

The  statistical  tests  used  to  analyze  the  data  were  the  Analysis  of  Variance 
(ANOVA),  two  sample  Mest,  and  the  nonparametric  Wilcoxon  test.  First  the  data  was 
analyzed  using  the  ANOVA  test  whether  overall  averages  for  a  specific  MOP  were 
statistically  similar  between  the  Janus  and  JLink  runs  taking  into  account  all  scenarios 
and  environments.  The  ANOVA  was  also  used  to  determine  interactions  between  the 
scenarios,  environments,  and  the  mode  (Janus  or  JLink)  for  each  MOP.  In  the  case  where 
the  ANOVA  determined  that  the  Janus  and  JLink  means  were  statistically  different,  pair¬ 
wise  two  sample  Mests  were  used  to  identify  where  the  differences  in  the  means  existed. 
Finally,  the  nonparametric  Wilcoxon  was  also  used  in  a  pair-wise  method  to  test  those 
populations  which  did  not  satisfy  Mest  assumptions  and  to  substantiate  the  Mest  results. 

The  results  of  the  analysis  show  that  JLink  requires  adjustments  to  artillery 
delivery  methods  in  order  to  calibrate  chemical  artillery  discrepancies  and  detection  range 
issues.  JLink  chemical  artillery  killed  far  fewer  entities  than  Janus,  and  JLink 
consistently  detected  farther  than  Janus.  The  finding  is  that  to  match  Janus,  JLink  must 
simulate  an  artillery  volley  as  one  large  cloud,  like  Janus,  as  opposed  to  individual 
rounds,  as  dictated  by  DIS  standards.  In  general,  JLink  accurately  portrays  coalition 
warfare  and  satisfactorily  replicates  armored  and  infantry  scenarios  in  contrasting 
environments. 
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I.  INTRODUCTION 


The  Army  relies  heavily  on  computer  simulations  for  training  and  force  analysis. 
One  simulation  in  use  today  is  Janus,  an  event-driven  wargame  named  for  the  two-faced 
Roman  god  Janus,  who  was  the  guardian  of  the  portals  of  Rome  and  the  patron  of 
beginnings  and  endings.  For  nearly  three  decades,  Janus  has  proven  to  be  a  valuable 
training  and  analysis  tool  and  has  evolved  into  a  legacy  model  with  many  validated 
features.  The  current  task  is  to  link  Janus,  which  is  designed  to  operate  in  a  stand-alone 
mode,  with  other  live,  virtual,  or  constructive  models. 

The  present  standard  used  to  connect  two  or  more  models  is  known  as  Distributed 
Interactive  Simulation  (DIS).  Janus  is  capable  of  operating  in  a  D1S  environment  using  a 
cell  adapter  unit  called  the  World  Modeler.  The  World  Modeler  is  software,  designed  to 
run  on  a  low-end  Silicon  Graphics  computer,  which  performs  functions  required  by  the 
DIS  architecture  that  are  not  performed  by  Janus.  The  combination  of  Janus  and  the 
World  Modeler  is  known  as  JLink  [Ref.  l:p.l], 

JLink  combines  the  training  and  analytic  potential  of  a  widely  fielded  constructive 
simulation,  Janus,  with  a  DIS-compatible  simulation.  A  goal  of  the  JLink  system  is  to  " 
replicate  the  analytical  and  training  fidelity  of  a  stand-alone  Janus  in  a  distributed  (JLink) 
exercise.  This  level  of  fidelity  may  be  gauged  by  comparing  the  results  of  a  scenario  run 
in  stand-alone  Janus  with  the  results  of  the  same  scenario  run  in  JLink,  using  similar 
starting  conditions.  The  purpose  of  this  thesis  is  to  assess  the  current  state  of  JLink 
development  by  measuring  the  amount  of  distortion  between  stand-alone  Janus  and 
distributed  Janus  using  JLink,  by  applying  statistical  analysis.  The  assessment  will  aid  in 
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identifying  credible  areas  of  JLink  as  well  as  aspects  of  JLink  which  require  further 
development.  As  interest  in  JLink  continues  to  grow  throughout  the  Army  and  across 
services,  improvements  in  JLink  fidelity  will  prove  valuable  and  return  many  training  and 
analytical  benefits. 

A.  JANUS 

1.  Janus  Background 

Lawrence  Livermore  National  Laboratory  originally  developed  Janus  in 
the  1970s,  with  the  current  version  developed  by  Training  and  Doctrine  Command 
Analysis  Center  (TRAC)  at  White  Sands  Missile  Range,  New  Mexico.  Janus  is  a 
stochastic,  interactive,  high  resolution,  multisided,  model  with  a  robust  database  and  a 
detailed  post-processor  for  gathering  results. 

Being  a  stochastic  model,  Janus  uses  a  random  number  generator  and 
probabilities  to  determine  the  outcome  of  detections  and  engagements.  Janus  random 
number  seeds  range  from  1  to  99.  Prior  to  executing  a  scenario,  the  user  may  set  the 
random  number  seed  manually  or  elect  to  have  Janus  randomly  select  the  seed.  Since 
Janus  is  an  event  based  model,  the  user  can  achieve  exact  replication  if  the  same  random 
number  seed  is  selected  and  there  is  no  human  interaction. 

The  interactive  nature  of  Janus  permits  its  users  to  make  decisions  during 
the  scenario  to  influence  the  outcome  of  the  battle.  Due  to  the  variability  of  user  inputs 
between  scenarios,  interaction  will  produce  different  results  for  two  Janus  scenarios  using 
an  identical  random  number  seed.  Examples  of  interactive  functions  include  creating  or 
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altering  movement  routes,  planning  and  firing  artillery  missions,  and 
mounting/dismounting  units.  User  interaction  is  not  necessary  for  pre-planned  scenarios 
designed  to  fight  two  opposing  forces.  For  purposes  of  this  thesis,  all  scenarios  were  pre¬ 
planned  and  included  no  interaction. 

Janus  is  a  high  resolution  model  because  it  represents  entities  down  to 
individual  systems.  However,  when  practical  the  user  may  aggregate  entities,  with 
artillery  systems  typically  being  aggregated  up  to  40  entities  [Ref.  2:p.21],  The  situation 
and  scenario  will  generally  dictate  the  level  of  aggregation,  with  Janus  users  often 
aggregating  to  the  lowest  useful  level  in  order  to  take  advantage  of  Janus’  high- resolution 
capabilities. 

Janus  can  also  model  up  to  six  different  sides  for  a  given  scenario,  to 
include  simulating  fratricide.  Janus  offers  the  option  to  separate  the  sides  among 
different  workstations  (primarily  for  training)  or  play  all  sides  on  one  workstation 
(primarily  for  analysis). 

Janus  possesses  a  robust  database  which  permits  the  user  to  define  a  weapon 
system  extensively  or  capture  the  detailed  factors  required  for  the  scenario.  The  database 
is  divided  into  six  main  sections:  systems,  weapons,  sensors,  engineer  data,  weather  data, 
and  chemical  and  heat  data.  Each  main  section  has  numerous  subsections,  all  requiring 
inspection  prior  to  creating  a  scenario  to  ensure  accurate  information  is  provided.  A 
separate  Janus  database  Manager’s  Manual  is  available  and  necessary  for  reference. 

Janus’  post-processor  permits  the  analyst  to  gather  detailed  results  from 
scenarios.  The  post-processor  offers  a  variety  of  detailed  statistics,  including  reports  on 
the  number  of  rounds  fired,  kill  ranges,  detection  ranges,  minefield  crossings,  chemical 
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casualties,  and  Loss  Exchange  Ratios  (LER).  The  user  has  the  option  to  select  which 
reports  will  appear  in  the  post-processor,  which  precludes  generating  unnecessary  reports 
while  focusing  on  the  area  of  interest.  The  reports  of  interest  for  this  thesis  are  those 
reports  mentioned  above. 

2.  Janus  Uses 

Originally  designed  as  an  analytical  model,  Janus  is  presently  used  in  the 
Training,  Exercises,  and  Military  Operations  (TEMO)  and  the  Advanced  Concepts 
Requirements  (ACR)  domains.  Both  domains  participate  as  members  of  the 
configuration  control  board  to  ensure  all  of  the  users’  requirements  are  satisfied. 

a.  Training 

Janus  is  best  geared  to  train  leaders  at  brigade  level  and  below 
because  of  its  high  resolution  capabilities.  The  ability  to  model  down  to  an  individual 
system  enables  scenarios  to  focus  on  company,  platoon,  or  squad  size  elements. 
Commanders  at  brigade  level  can  use  Janus  to  train  the  leaders  of  these  elements  in  the 
decision-making  process.  Janus’  interactive  capabilities  allow  leaders  to  make  decisions 
during  an  engagement,  and  then  to  observe  the  outcome  of  their  actions.  Following  the 
scenario,  the  entire  battle  can  be  replayed  using  the  Janus  Analyst  Workstation  (JAWS) 
and  assessed  for  after-action  review  purposes. 
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b.  Analysis 


Janus  is  particularly  suited  for  the  analyst.  Janus’  post-processor 
permits  the  analyst  to  gather  results  from  scenarios  for  further  study  of  topics  such  as  new 
tactics,  techniques,  and  procedures.  The  analyst  creates  scenarios  using  standard 
methods,  and  then  compares  the  results  to  a  scenario  using  newly  proposed  methods. 

The  analyst  also  uses  Janus  to  study  the  effects  of  a  new  or  modified 
weapon  system.  Janus’  database  is  robust  enough  to  capture  or  alter  many  aspects  of  a 
weapon  system,  including  weapon  range,  detection  range,  sensor  type,  travel  speeds,  and 
crew  size.  These  weapon  systems  can  also  be  tested  in  various  environments  and  weather 
conditions  based  on  the  weapon’s  mission  profile. 

As  a  stand-alone  simulation,  Janus  has  aided  Army  trainers  and  analyst  for 
years.  Janus  must  now  move  to  the  next  level  of  simulation,  the  DIS  environment,  in 
order  to  remain  beneficial  to  the  Army  of  the  21st  century. 

B.  DISTRIBUTED  INTERACTIVE  SIMULATION  (DIS) 

1.  DIS  Background 

The  DIS  Master  Plan  defines  DIS  as  “A  synthetic  environment 
within  which  humans  may  interact  through  simulation(s)  and/or  simulators  at  multiple 
networked  sites  using  compliant  architecture,  modeling,  protocols,  standards,  and 
databases”  [Ref.  3:p.  1-1].  Leading  edge  computer  technologies  and  advanced 
communications  networking  allow  for  such  an  environment  to  exist.  Essentially,  DIS  is 
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an  environment  which  brings  the  power  of  the  computer  and  communications  together  to 
benefit  the  trainer  and  analyst. 

The  terms  of  DIS  are  defined  as  follows:  Distributed  refers  to  geographically 
separated  simulations,  each  hosted  on  a  computer  and  connected  via  networks  to  create  a 
shared  environment.  Interactive  is  described  as  different  simulations  electronically  linked 
to  act  together  and  upon  one  another.  Simulation  is  divided  into  three  categories:  a)  live 
simulation  where  actual  equipment  and  soldiers  are  operating  in  the  field,  b)  constructive 
simulation  which  includes  wargames  and  models  such  as  Janus,  and  c)  virtual  simulation 
which  involves  manned  simulators  interacting  within  a  virtual  reality  environment  [Ref. 
3:p.  1-2  - 1-3]. 

The  primary  objective  of  the  DIS  program  is  to  establish  an  architecture  which 
permits  the  linkage  of  different  simulation  environments  into  a  seamless  synthetic  world. 
The  infrastructure  brings  together  systems  built  for  separate  purposes,  technologies  from 
different  eras,  products  from  different  vendors,  and  platforms  from  various  services  [Ref. 
3:p.  1-2].  Janus,  operating  in  a  DIS  environment  at  Ft.  Hood,  Texas,  linked  to  a  virtual 
AH-64  Apache  simulator  at  Ft.  Hood  and  a  constructive  ModSAF  simulation  at  Ft. 

Knox,  Kentucky  is  a  realistic  application  of  the  DIS  objective  (described  above). 

2.  DIS  Benefits 

Since  the  development  of  DIS  in  1991  with  the  successful  linking  and 
interacting  of  the  Simulation  Network  (SIMNET)  program,  observers  have 
enthusiastically  embraced  the  benefits  of  DIS.  Specifically,  a  DIS  environment  can 


6 


benefit  the  Army  and  other  services  in  the  areas  of  training,  combat  development,  and 
systems  acquisition. 

In  terms  of  training,  commanders  have  greater  potential  to  train  as  they  fight.  If  a 
DIS  capability  is  established  between  live  and  constructive  and/or  virtual  simulations, 
training  will  more  realistically  mirror  combat  conditions.  Another  aspect  of  DIS  training 
benefits  is  in  the  area  of  jointness.  An  objective  of  DIS  is  to  link  simulations  across 
services.  Such  a  link  enhances  joint  training  and  promotes  increased  joint  training 
activities. 

DIS  provides  the  combat  development  community  expanded  capabilities.  First, 
DIS  provides  the  ability  to  introduce  new  simulations  into  the  synthetic  DIS  environment, 
such  as  an  individual  soldier  simulated  as  a  virtual  entity,  with  minimal  impact  on 
software.  DIS  also  provides  combat  developers  the  capability  to  collect  and  record 
actions  and  reactions  generated  by  humans  with  minimal  interference  from  observers 
[Ref.  3:p.  HI-4]. 

The  benefit  DIS  brings  to  systems  acquisition  is  reduced  test  and  evaluation  costs. 
By  linking  live,  constructive,  and  virtual  simulations,  the  acquisition  life  cycle  can  be 
reduced  significantly,  resulting  in  test  and  evaluation  cost  reductions. 

The  focus  of  this  thesis  is  evaluating  Janus  in  a  DIS  environment.  The  value  of 
Janus  and  the  importance  of  DIS  were  the  driving  forces  in  the  development  of  JLink,  the 
method  by  which  Janus  operates  in  the  DIS  environment.  The  next  section  highlights  the 
methods  of  JLink  and  the  challenges  faced  by  JLink. 
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C.  JLINK 


1.  JLink  Background 

In  1993,  TRAC  Monterey  began  developing  the  JLink  system  to  support  the  Anti- 
Armor  Advanced  Technology  Demonstration  (A2ATD).  The  A2ATD  was  charged  with 
developing  and  demonstrating  the  use  of  DIS  to  evaluate  anti-armor  weapon  systems  on  a 
combined  arms  task  force.  TRAC  Monterey  had  the  overall  responsibility  for  building 
the  systems,  providing  project  management,  and  contributing  functional  area  expertise. 
Other  contributors  to  the  JLink  project  include  the  Naval  Postgraduate  School  (NPS), 
RAND,  MITRE,  and  Rolands  and  Associates  (R&A).  NPS  and  RAND  developed  the 
interface  between  Janus  and  the  World  Modeler.  RAND  also  conducted  terrain 
conversions  and  modified  Janus  internal  algorithms  as  necessary,  while  MITRE  focused 
on  implementing  dead  reckoning  algorithms.  [Ref.  l:p.  1] 

Following  the  A2ATD,  TRAC  Monterey  continued  to  develop  JLink  to  serve  the 
TEMO  and  ACR  domains  with  future  training  and  analytical  requirements.  To  assess  the 
state  of  JLink  development,  TRAC  Monterey  conducted  a  comparison  study  of  Janus  to 
JLink  in  the  summer  of  1996.  The  study  applied  a  rudimentary  scenario  in  a  single 
environment  to  test  basic  engagement  methodologies.  The  results  of  the  study  showed 
that  JLink  produced  similar  kill  ranges  and  loss  exchange  ratios  when  compared  to  Janus, 
but  statically  different  shot  totals  for  the  blue  forces.  The  cause  of  the  total  shots 
discrepancy  was  identified  as  a  z-coordinate  issue  with  respect  to  aircraft,  and  the  study 
concluded  that  a  scenario  fought  in  stand  alone  Janus  is  “closely  matched”  with  the  same 
scenario  fought  under  JLink’s  current  configuration  [Ref.  4:p.  9-10]. 
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As  TRAC  continues  to  develop  JLink,  recurrent  testing  is  necessary  to  assess 
JLink’s  current  state.  As  JLink  evolves  into  a  robust  model,  capable  of  simulating 
multiple  battlefield  phenomena,  comparison  studies  between  Janus  and  JLink  must  be 
diverse  enough  to  test  and  analyze  newly  developed  aspects  of  the  system.  This  thesis 
conducts  such  a  test  by  simulating  large  scale  scenarios  in  multiple  environments.  The 
experiment  also  includes  recently  developed  JLink  functions  such  as  Family  of 
Scatterable  Mines  (FASCAM)  and  chemical  artillery. 

2.  JLink  Architecture 

This  section  describes  the  JLink  architecture,  which  enables  Janus  to 
operate  in  a  DIS  environment  through  use  of  the  World  Modeler.  Network  management 
establishes  the  connection  between  Janus  and  other  DIS-compatible  models,  such  as 
ModSAF,  using  two  different  protocols.  The  Transmission  Control  Protocol/Internet 
Protocol  (TCP/IP)  is  used  to  establish  communications  between  the  World  Modeler  and 
Janus,  while  the  User  Datagram  Protocol  (UDP/IP)  is  used  to  establish  communications 
between  the  World  Modeler  and  DIS.  Through  these  links,  the  World  Modeler  controls 
dataflow  by  passing  Protocol  Data  Unit  (PDU)  information  to  Janus  and  DIS  at  the 
appropriate  time  intervals  [Ref.  l:p.  5]  (Figure  1).  The  purpose  of  the  PDU  is  to 
facilitate  the  electronic  transfer  of  data  between  simulations  with  different  software. 
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Suppose  Janus  is  portraying  Blue  forces  and  a  DIS-compatible  simulation  is 


portraying  Red  forces.  When  Blue  generates  an  event,  such  as  firing  at  a  Red  entity, 
Janus  sends  a  message  to  the  World  Modeler  via  the  TCP/IP  connection.  The  World 
Modeler  uses  “hooks”  to  capture  the  Janus  protocol  and  converts  the  Janus  protocol  to  a 
DIS  PDU,  which  can  he  processed  by  a  DIS  model.  The  World  Modeler  then  broadcasts 
the  DIS  PDU  via  the  UDP/IP  connection  to  the  network  for  use  by  the  DIS  (Red) 
simulation.  Conversely,  when  the  Red  force  generates  an  event,  the  corresponding  DIS 
PDU  is  sent  via  the  network  to  the  World  Modeler,  where  it  converts  the  DIS  PDU  to  a 
Janus  protocol,  which  in  turn  is  sent  to  the  Janus  simulation  for  processing.  In  essence. 
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one  of  the  functions  of  the  World  Modeler  is  to  convert  Janus  Protocols  to  DIS  PDUs  and 
vice  versa. 

Currently,  JLink  has  successfully  interacted  with  several  constructive  simulations 
and  virtual  simulators,  via  DIS  PDUs.  However,  evolving  systems  require  extensive 
testing,  and  the  JLink  process  frequently  requires  adjustments  or  model  calibration  to 
achieve  effective  levels  of  interoperability  between  interfacing  systems. 
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II.  PROBLEM  DESCRIPTION 


Improved  interoperability  can  be  translated  into  minimizing  the  distortion  between 
results  from  Janus  stand-alone  and  the  results  of  the  same  scenario  run  in  JLink.  Two  or 
more  simulations/simulators  are  defined  as  “DIS  interoperable”  when  their  performance 
characteristics  support  a  fair  fight  to  the  fidelity  required  for  the  exercise.  A  fair  fight 
exists  when  the  differences  in  two  simulations’  performance  are  overwhelmed  by  user 
actions  [Ref.  5:p.  A-4],  There  are  two  general  factors  which  may  contribute  to 
differences  in  the  results  between  Janus  stand-alone  and  JLink  scenarios  and  lead  to  an 
unfair  fight.  These  factors  are  the  JLink  architecture  and  the  DIS  PDU  itself,  which  form 
the  basis  for  the  problem  description  of  this  thesis.  This  chapter  addresses  how  JLink 
architecture  and  the  DIS  PDU  may  contribute  to  distortion.  As  part  of  the  problem 
description,  the  chapter  concludes  by  identifying  specific  features  of  JLink,  which  may  be 
affected  by  the  factors  mentioned  above,  that  require  testing. 

A.  DISTORTION  FROM  JLINK  ARCHITECTURE 

Inherent  aspects  of  the  JLink  architecture  may  contribute  to  differences  in 
JLink  results  versus  Janus  stand-alone  results.  Given  that  Janus  is  an  event-driven  model, 
the  sequence  of  events  is  vital  to  the  outcome.  In  stand-alone  Janus,  a  scenario  will 
produce  the  same  sequence  of  events,  and  hence  the  same  results,  for  every  run  as  long  as 
the  random  number  seed  remains  unchanged  and  no  human  interaction  occurs.  In  JLink, 
delays  in  passing  and  converting  PDUs  may  produce  a  different  sequence  of  events  than 
the  corresponding  Janus  stand-alone  scenario.  Delays  in  passing  PDUs  between  DIS 
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models  are  a  function  of  network  congestion  and  quality  of  the  network  hardware. 

Delays  in  converting  Janus  protocols  to  DIS  PDUs,  and  vice  versa,  also  contribute  to  an 
altered  sequence  of  events.  A  different  sequence  of  events  will  result  in  different 
outcomes.  Say  for  instance  a  Blue  entity  in  stand-alone  Janus  generates  a  shot  event  on 
the  event  list  against  a  Red  entity  at  time  =  10  minutes,  resulting  in  a  Red  kill  at  time  =  1 1 
minutes.  In  JLink,  the  same  shot  event  may  not  get  scheduled  until  time  =  12  minutes 
due  to  delays  in  passing  and  converting  the  appropriate  PDU,  which  may  result  in  the  Red 
entity  surviving  (and  shooting)  longer  than  in  Janus  stand-alone. 

Another  aspect  of  JLink  architecture  contributing  to  distortion  is  the  issue  of 
random  number  generators.  Janus  stand-alone  utilizes  only  one  random  number 
generator  to  determine  the  outcome  of  all  stochastic  events.  JLink  uses  the  Janus  random 
number  generator  to  adjudicate  only  those  events  corresponding  to  its  own  side.  The 
other  sides  represented  in  the  scenario  use  their  own  random  number  generators  to 
adjudicate  stochastic  events.  Even  if  all  random  number  generators  involved  behave 
identically  (which  is  unlikely)  and  they  are  all  initiated  with  the  same  random  number 
seed,  they  will  not  produce  the  same  aggregate  random  number  string  as  the  Janus  stand¬ 
alone  generator  for  an  entire  scenario.  Dissimilar  random  number  strings  will  result  in 
dissimilar  results  between  Janus  stand-alone  and  JLink. 

This  thesis  will  not  attempt  to  determine  whether  any  JLink  architecture  issues 
like  those  above  contributed  to  distorted  results.  The  thesis  experiment  did,  however, 
reduce  the  impact  of  the  architecture  on  results  by  using  the  following  procedures.  To 
reduce  interference  when  passing  PDUs,  all  JLink  runs  were  executed  during  times  when 
the  TRAC  Monterey  network  was  free  from  all  other  activity.  Also,  all  scenarios  run  in 
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JLink  used  the  same  random  number  seeds  used  in  Janus  stand-alone.  Using  the  same 
random  number  seeds  will  produce  some  of  the  same  random  numbers  for  both  JLink  and 
Janus  stand-alone  scenarios. 

B.  DISTORTION  FROM  DIS  PDUs 

One  major  challenge  to  World  Modeler  developers  is  to  accurately  translate  and 
create  DIS  PDUs  as  they  flow  to  and  from  the  Janus  model.  The  DIS  PDUs  created  by 
the  World  Modeler  for  use  by  other  distributed  systems  must  portray  the  originating  Janus 
protocol  as  accurately  as  possible.  Likewise,  DIS  PDUs  sent  to  the  World  Modeler  by 
distributed  simulations  must  be  translated  appropriately  for  use  by  Janus.  Inaccurate  or 
inappropriate  conversions  of  PDUs  may  produce  distorted  JLink  results  as  compared  to 
the  Janus  stand-alone  standard. 

One  example  where  appropriate  conversions  of  PDUs  is  critical  is  the  flight 
altitude  of  rotary-wing  aircraft.  Janus  only  portrays  two  flight  levels,  treetop  level  (low) 
and  at  altitude  (high),  whereas  ModSAF  simulates  continuous  altitudes.  The  issue 
becomes  at  which  flight  level  a  ModSAF  helicopter  entity  should  be  simulated  in  Janus, 
and  at  what  altitude  a  low  flying  Janus  helicopter  should  be  simulated  in  ModSAF. 

In  order  to  test  for  distortion  caused  by  the  DIS  PDUs,  JLink  requires  additional 
testing  of  a  wide  variety  of  features.  To  date,  JLink  fidelity  has  only  been  tested  using 
rudimentary  scenarios  in  a  single  environment.  JLink  development  is  now  at  the  stage 
which  can  support  testing  of  additional  aspects  of  the  system. 


c. 


JLINK  FEATURES  TESTED  IN  EXPERIMENT 


1.  Varied  Scenarios 

One  aspect  of  the  JLink  system  this  thesis  tests  is  JLink’s  ability  to 
accurately  portray  different  types  of  combat  units  in  the  offense  or  defense,  negotiating 
obstacles,  under  different  environmental  conditions.  Such  scenarios  have  yet  to  be 
validated  using  JLink.  If  JLink  operates  as  designed,  the  results  from  these  varied 
scenarios  will  be  similar  to  the  results  of  the  same  scenarios  run  in  stand-alone  Janus. 

2.  Coalition  Warfare 

The  thesis  experiment  also  includes  an  assessment  of  JLink  simulating  the 
synergistic  effects  of  coalition  warfare.  Previous  tests  involving  JLink  have  only 
included  two  sides  fighting  against  one  another.  As  part  of  the  experiment,  one  scenario 
includes  three  sides,  two  forces  allied  together  fighting  against  the  third.  Pitting  two  sides 
against  a  lone  third  side  may  generate  insightful  results  regarding  JLink’s  ability  to 
accurately  fight  a  coalition  battle. 

3.  Specific  Combat  Functions 

In  addition,  the  thesis  tests  whether  JLink  accurately  portrays  the  effects  of 
specific  combat  functions.  Recent  JLink  software  inputs,  designed  to  replicate  specific 
combat  functions  such  as  chemical  artillery  delivery  methods  and  FASCAM,  require 
testing.  In  stand-alone  Janus,  chemical  artillery  volleys  are  delivered  as  an  aggregated 
threat,  with  one  large  radius.  DIS  standards  require  that  individual  artillery  rounds  be 
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passed  between  simulations.  Because  of  this  requirement,  programmers  have  modeled  a 
JLink  chemical  artillery  volley  as  having  each  round  create  its  own  radius  of  effect. 
Figure  2  depicts  a  volley  of  three  artillery  rounds  and  illustrates  the  difference  in  the  two 
delivery  methods. 


Figure  2.  Artillery  Delivery  Methods 


The  thesis  will  determine  whether  such  a  distribution  of  chemical  artillery  rounds 
significantly  affects  the  number  of  kills. 

Another  specific  combat  function  requiring  testing  is  FASCAM,  an 
artillery-delivered  minefield.  Unlike  chemical  artillery,  aggregation  is  not  an  issue 
regarding  FASCAM.  This  thesis  provides  initial  feedback  on  the  FASCAM  code  as  to 
whether  it  generates  the  same  number  of  kills  as  FASCAM  in  stand-alone  Janus. 

Herein  lies  the  goal  of  this  thesis  —  to  assess  JLink  fidelity  by  testing  whether 
JLink  produces  similar  results  to  stand-alone  Janus  using  varied  scenarios  and  coalition 
warfare,  and  by  determining  whether  recent  software  inputs  to  specific  combat  functions 
are  providing  accurate  representations. 
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III.  DESIGN  OF  EXPERIMENT 


The  overall  goal  of  the  experiment  was  to  provide  a  means  to  obtain  data  from  a 
scenario  executed  in  Janus  and  the  same  scenario  executed  in  JLink  for  purposes  of 
analysis.  The  issues  involved  in  meeting  the  goal  were  selecting  appropriate  data  to  be 
analyzed,  determining  the  structure  of  the  scenarios,  and  conducting  the  experiment. 

This  chapter  begins  by  presenting  the  measures  of  performance  and  the  methodology 
supporting  their  selection.  Next,  details  of  the  scenarios  are  described,  followed  by  a 
discussion  of  the  conduct  of  the  experiment  and  issues  encountered  with  the  execution. 
The  experiment  included  necessary  conditions  for  testing  the  issues  defined  in  Chapter  II. 

A.  MEASURES  OF  PERFORMANCE 

The  first  issue  in  designing  the  experiment  was  to  identify  what  data  would  be 
analyzed.  The  MOPs  were  limited  only  by  the  information  provided  in  the  post¬ 
processor.  For  clarity,  this  thesis  refers  to  the  data  as  measures  of  performance,  as 
opposed  to  measures  of  effectiveness.  The  reason  for  the  distinction  is  that  the  purpose  of 
the  thesis  is  not  to  analyze  the  effectiveness  of  a  new  weapon  system  or  perhaps  a  tactic. 
Rather,  the  focus  is  to  assess  JLink  fidelity  by  comparing  the  performance  of  the  JLink 
system  to  the  performance  of  the  stand-alone  Janus  system. 

The  MOPs  were  derived  from  issues  raised  in  the  Problem  Description.  The  areas 
that  require  testing  in  JLink  are  varied  scenarios,  meaning  scenarios  using  different  types 
of  units  in  contrasting  environments,  coalition  warfare,  and  specific  combat  functions 
such  as  chemical  artillery  and  FASCAM.  The  determination  whether  JLink  accurately 
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portrays  the  areas  defined  in  the  Problem  Description  is  derived  from  the  analysis  of  one 
or  more  of  the  following  MOPs:  1)  Detection  Ranges,  2)  Kill  Ranges,  3)  Rounds  Fired, 
4)  Loss  Exchange  Ratio  (LER),  5)  Chemical  Artillery  Kills,  and  6)  FASCAM  Kills. 

1.  MOPs  for  Varied  Scenarios  and  Coalition  Warfare 

To  determine  the  fidelity  of  varied  scenarios  in  JLink,  the  thesis  analyzed 
detection  ranges,  kill  ranges,  rounds  fired,  and  LER.  Applying  appropriate  MOPs  to  the 
broad  topic  of  scenario  fidelity  is  critical  to  arriving  at  valid  conclusions.  The  analysis 
must  focus  on  the  basic  procedures  of  an  engagement.  First,  one  entity  must  detect 
another  entity.  Comparing  the  detection  ranges  of  Janus  stand-alone  and  JLink  is  a 
quantitative  method  of  assessing  the  detection  process  in  JLink.  Next,  the  detecting  entity 
may  elect  to  shoot  at  the  target,  possibly  resulting  in  a  kill.  Analyzing  the  kill  ranges 
between  Janus  and  JLink  is  a  quantitative  method  for  assessing  the  killing  process  in 
JLink.  Since  not  all  shots  result  in  kills,  the  total  number  of  rounds  fired  must  also  be 
analyzed.  The  post-processor  also  provides  the  flexibility  to  analyze  detection  ranges,  kill 
ranges,  and  rounds  fired  for  an  entire  scenario  or  by  individual  side.  For  instance,  rounds 
fired  can  be  analyzed  by  comparing  total  Janus  rounds  fired  to  total  JLink  rounds  fired  for 
the  same  scenario,  or  by  comparing  blue  Janus  rounds  versus  blue  JLink  rounds. 

Detecting  and  killing  the  target  are  the  basic  elements  of  a  battle.  If 
JLink  replicates  these  functions  in  the  likeness  of  Janus,  one  would  expect  similar  overall 
outcomes  between  Janus  stand-alone  and  JLink  scenarios.  Further,  since  LERs  are 
primarily  a  function  of  detections  and  kills,  the  LERs  offer  a  quantitative  method  to 
assess  the  overall  outcome  of  a  JLink  scenario. 
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2. 


MOPs  for  Chemical  Artillery  and  FASCAM 


To  determine  whether  JLink  achieves  acceptable  results  from  chemical 
artillery  and  FASCAM,  the  thesis  analyzed  the  number  of  kills  generated  by  chemical 
artillery  and  the  number  of  kills  resulting  from  firing  FASCAM. 

B.  SCENARIOS 

The  scenarios  were  determined  based  on  the  Problem  Description  requirements. 
Specifically,  the  thesis  must  analyze  results  from  varied  scenarios,  coalition  warfare,  and 
the  specific  combat  functions  of  chemical  artillery  and  FASCAM.  US  and  Russian-style 
armored  and  light  infantry  units,  along  with  their  supporting  elements,  were  a  logical 
selection  for  the  unit  types.  Armored  and  light  infantry  units  are  combat  arms  likely  to  be 
involved  in  offensive  or  defensive  operations.  Units  equipped  with  US-  and  Russian- 
made  weapons  bring  diverse  capabilities  to  the  battlefield,  adding  robustness  to  the  study. 

In  determining  the  environments,  the  goal  was  to  select  two  contrasting  locations 
which  will  likely  have  different  effects  on  detection  ranges  and  engagement  ranges.  The 
two  contrasting  environments  used  were  Southwest  Asia  (SWA)  and  Hunter  Liggett, 
California  (HL).  SWA  is  primarily  desert  with  no  trees  and  slightly  undulating  terrain, 
which  provides  some  relief.  HL  is  comprised  of  small  hills  and  lightly  wooded  terrain. 
The  relatively  flat  terrain  of  SWA  compared  to  the  rolling,  wooded  terrain  of  HL  offers 
contrasting  settings  to  test  sensor  functions  and  weapons  platforms. 

After  determining  the  types  of  units  and  environments,  the  final  element  in  the 
scenario  structure  was  the  means  of  engagement.  The  engagement  used  was  an  attacking 
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battalion  against  a  defending  company.  Such  an  engagement  provided  results  from  units 
with  contrasting  postures,  moving  as  opposed  to  stationary.  Different  postures  affect  the 
detection  and  firing  capabilities  of  entities  in  Janus. 

Once  all  of  the  elements  of  the  scenarios  were  identified,  three  scenarios  were 
developed.  The  first  scenario  involved  a  US-equipped  armored  company  (blue) 
defending  against  an  attacking  Russian-equipped  armored  battalion  (red).  The  second 
scenario  was  identical  to  the  first,  except  that  the  red  force  was  divided  into  two  sides  to 
form  a  coalition.  The  red  coalition  attacked  the  (defending)  blue  force  to  test  the  results 
of  a  coalition  engagement,  in  both  a  stand-alone  and  distributed  mode.  The  third  scenario 
involved  a  US-equipped  light  infantry  company  (blue)  defending  against  an  attacking 
Russian-equipped  light  infantry  battalion  (red).  In  every  scenario,  the  blue  side  fired 
chemical  artillery  rounds  and  FASCAM  at  specifically  identified  red  units  traveling  on  a 
narrow  avenue  of  approach.  This  was  done  to  isolate  the  weapons’  performance  to  aid  in 
the  deconfliction  of  MOPs  for  analysis.  Also,  all  blue  sides  possessed  obstacles  to  aid  in 
the  defense.  The  obstacles  included  minefields  and  abatis  for  road  barriers. 

1.  Scenario  1 

The  weapon  systems  used  by  the  blue  side  in  scenario  1  were  the  MIA  1 
tank,  M2  Infantry  Fighting  Vehicle  (IFV),  Multiple  Launcher  Rocket  System  (MLRS) 
artillery,  AH64  Apache  helicopter,  and  four  riflemen  as  scouts.  The  red  side  consisted  of 
the  T72  tank,  BMP-2  IFV,  MLR  16  (MLRS  equivalent),  Hind  helicopter,  and  SAM  air 
defense  weapon.  The  blue  side  consisted  of  32  total  units  (mainly  MlAls),  while  the  red 
side  had  91  total  units  (mainly  T72s),  resulting  in  approximately  a  3:1  ratio  in  favor  of  the 
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attacker.  The  scenario  was  designed  to  emphasize  the  capabilities  of  the  M1A1  and  T72 
tanks,  with  support  from  the  other  systems. 

2.  Scenario  2 

The  weapon  systems,  weapon  numbers,  and  scenario  design  in  scenario  2 
were  identical  to  those  in  scenario  1.  In  scenario  2,  the  red  force  was  divided  into  two 
sides,  fighting  together  as  a  coalition  against  the  blue  force. 

3.  Scenario  3 

The  weapon  systems  used  by  the  blue  side  in  scenario  3  were  the  rifleman  with 
Ml 6  rifle,  rifleman  with  machine  gun.  Light  Anti-tank  Weapon  (LAW),  mortars,  and 
MLRS.  The  red  side  consisted  of  the  rifleman  with  AK47  rifle,  rifleman  with  machine 
gun,  anti-tank  RPG,  mortars,  and  MLR16.  The  blue  side  had  1 16  units  versus  653  units 
for  the  red  side,  resulting  in  a  5.6: 1  edge  in  favor  of  the  attacker.  The  scenario 
emphasized  the  capabilities  of  the  rifle  and  machine  gun. 

4.  Scenario  General  Characteristics 

Red  forces  attacked  from  north  to  south  in  all  scenarios  since  likely  avenues  of 
approach  in  both  environments  favored  north-south  attack  routes.  Recall  that  blue  forces 
fired  chemical  rounds  at  specific  red  units  to  assess  their  effectiveness  in  distributed 
simulations.  Since  the  wind  generally  travels  from  southeast  to  northwest  in  both 
environments,  it  was  logical  to  have  the  blue  side  firing  northward,  or  downwind.  Hence, 
the  red  forces  were  arrayed  to  attack  south. 
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Another  common  characteristic  between  scenarios  was  the  placement  of  the  blue 


forces.  All  blue  units  were  placed  on  relatively  defensible  terrain,  oriented  to  the  north. 
Ensuring  the  blue  units  had  adequate  Line  of  Sight  (LOS)  to  the  north  was  critical  to  the 
defense.  Figure  3  shows  the  Janus  map  of  HL,  with  the  blue  units  in  position.  The  figure 
also  shows  the  LOS  of  one  blue  entity,  with  “blind  spots”  shown  as  broken  radials. 


Figure  3.  Scenario  Graphics 


Additionally,  the  scenarios  were  designed  so  that  all  blue  entities  could  fire  at  all 


red  entities,  and  vice  versa.  This  design  required  slight  adjustments  to  the  database  in 
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some  instances.  The  purpose  of  this  design  was  to  test  JLink’s  ability  to  portray  multiple 
types  of  weapons  engaging  each  other. 

Finally,  all  scenarios  lasted  between  17-27  minutes,  with  the  average  length  being 
approximately  20  minutes.  Although  the  test  scenario  is  short  compared  to  typical  Janus 
exercises,  which  often  last  several  days,  20  minutes  permitted  artillery  units  to  fire 
assigned  volleys,  “pucked”  FASCAM  to  fire  five  minefields,  aircraft  to  fly  designated 
routes,  and  attacking  elements  to  move  into  and  through  the  engagement  area. 

C.  EXPERIMENT 

1.  Conduct  of  Experiment 

Each  of  the  three  scenarios  were  executed  in  two  environments,  SWA  and  HL, 
resulting  in  six  combinations.  Each  combination  was  then  run  in  two  different  modes. 
First  they  were  run  in  the  Janus  stand-alone  mode,  Janus  (blue)  versus  Janus  (red),  with 
both  the  blue  and  red  forces  simulated  on  the  same  Hewlett-Packard  (HP)  workstation. 
Then  the  same  six  combinations  were  run  in  the  JLink  (blue)  versus  JLink  (red)  mode, 
with  the  two  forces  simulated  on  separate  workstations,  as  illustrated  in  Figure  4.  The 
JLink  versus  JLink  mode  is  the  method  developed  at  TRAC  Monterey  to  test  distributed 
Janus  without  having  to  connect  to  another  type  of  DIS  simulation.  Each  Janus  terminal 
is  connected  to  a  World  Modeler  which  in  turn  is  connected  to  the  network. 
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JUNK 


Figure  4.  JLinkvsJLink 

Six  combinations  run  in  two  different  modes  resulted  in  12  total  combinations  in 
order  to  capture  all  scenarios,  environments,  and  modes.  Figure  5  depicts  the  overall 
experiment. 


Janus  Stand-Alone 

JLinkvsJLink 

Armored  SWA 
Armored  HL 

(avg  of  10  runs) 
(avg  of  10  runs) 

(avg  of  10  runs) 
(avg  of  10  runs) 

Armored  Coalition  SWA 
Armored  Coalition  HL 

(avg  of  10  runs) 
(avg  of  10  runs) 

(avg  of  10  runs) 
(avg  of  10  runs) 

Light  Infantry  SWA 
Light  Infantry  HL 

(avg  of  10  runs) 
(avg  of  10  runs) 

(avg  of  10  runs) 
(avg  of  10  runs) 

Figure  5.  Overall  Experiment  (Averages  for  Specific  MOP) 


These  12  combinations,  however,  only  provide  one  data  point  for  each 
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MOP,  and  will  be  referred  to  as  a  single  run  of  the  experiment.  Ultimately,  10  runs  were 
performed  for  each  cell  to  provide  a  reasonable  sample.  (The  issue  regarding  the  total 
number  of  runs  is  addressed  in  Chapter  III,  paragraph  C.  2.) 

The  desired  method  of  analysis  was  to  compare  the  MOPs  resulting  from  a 
specified  number  of  Janus  stand-alone  runs  to  the  MOPs  resulting  from  the  same  number 
of  JLink  runs.  Since  a  stand-alone  run  will  produce  the  exact  same  results  if  the  same 
random  number  seed  is  sown,  all  stand-alone  runs  were  executed  using  different, 
randomly  selected,  seeds.  The  same  random  number  seeds  were  then  used  in  the 
corresponding  JLink  runs.  Also,  due  to  the  inherent  variability  caused  by  JLink 
architecture,  10  additional  runs  were  performed  in  JLink  using  the  same  random  number 
seed  for  all  10  runs  to  isolate  the  variance  contribution  from  that  aspect  of  the  JLink 
system. 


2.  Experiment  Issues 

As  mentioned  previously,  all  JLink  runs  were  done  during  times  of  minimal  traffic 
on  the  TRAC  Monterey  network.  In  all  but  four  JLink  runs,  the  scenarios  were  executed 
when  the  network  was  completely  free  of  other  users.  Executing  the  scenarios  with 
minimal  network  interference  reduces  variability  in  the  results  due  to  event  scheduling.  If 
the  network  is  congested,  delays  may  occur  in  passing  information  and  the  sequence  of 
events  may  be  affected. 

Determining  the  sample  size  is  fundamental  to  the  design  of  every  experiment. 

An  experiment  is  conducted  a  number  of  times  so  that  the  data  produces  good  estimators 
of  the  true  population  parameters.  Different  techniques  are  available  to  arrive  at  a 
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satisfactory  sample  size.  The  thesis  proceeded  as  follows  regarding  this  issue.  First,  a 
practical  approach  was  used.  Based  on  experience  and  recommendations  from  senior 
Janus  analysts,  a  sample  size  of  n  =  10  generally  produces  results  with  “acceptable 
variance”  for  the  MOPs  defined  in  this  thesis.  That  is,  the  experiment  is  run  until  an 
estimate  of  the  variance  for  the  mean  is  reduced  to  a  pre-determined,  acceptable  level. 
This  level  will  be  different  for  each  measure  of  performance.  Based  on  the  raw  data 
figures  in  Appendix  A,  acceptable  variance  was  achieved  in  detection  ranges,  kill  ranges, 
LERs,  and  for  most  cases  of  rounds  fired.  Discussion  in  Chapter  IV  highlights  anomalies 
in  FASCAM  and  chemical  kills  which  preclude  the  possibility  of  reaching  acceptable 
variances  levels  for  these  MOPs  with  reasonable  sample  sizes. 

The  thesis  also  considered  the  issue  of  normality  of  the  data  when 
determining  sample  size.  Normality  of  the  data  set  is  one  assumption  in  the  two  sample  t- 
test,  which  is  used  for  analysis  of  the  MOPs.  The  normal  distribution  quantile  plots  in 
Appendix  F  show  that  the  results  for  most  MOPs  are  relatively  normal,  with  some 
obvious  exceptions.  Also,  the  t-test  normality  assumption  is  generally  satisfied  if  the 
sample  averages  are  normally  distributed.  As  the  sample  size  increases,  one  would 
expect  the  sample  averages  to  become  more  normal,  based  on  the  Central  Limit  Theorem 
[Ref.  6:p.  232],  In  this  case,  we  expect  a  sample  size  of  n=10  to  be  large  enough  to 
produce  normally  distributed  averages. 

Another  issue  addressed  prior  to  beginning  the  runs  was  that  the  user  can  only  fire 
FASCAM  while  the  scenario  is  on-going.  The  users  timing  and  accuracy  could  vary 
significantly  from  run  to  run  and  thus  alter  the  FASCAM  results.  The  solution  was 
discovered  in  the  “puck”  run.  In  essence,  a  puck  run  records  all  of  the  actions  of  a 
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previously  run  scenario,  including  the  user  actions,  and  replays  them  on  the  current 
scenario  run.  By  pucking  the  initial  run  of  each  scenario,  then  changing  the  random 
number  seeds  for  the  following  runs,  the  user  can  replicate  the  same  scenario,  to  include 
FASCAM  volleys,  while  generating  different  results  due  to  the  new  seed. 

The  final  issue  encountered  in  conducting  the  experiment  occurred  after  the  runs 
were  complete.  The  issue  involved  gathering  the  information  from  the  post-processor 
into  a  usable  form.  After  transferring  the  post-processor  information  to  a  personal 
computer,  extensive  manipulation  of  the  data  was  required  prior  to  importing  the  data 
into  a  spreadsheet  or  statistical  package.  Since  the  results  from  the  post-processor  are 
lengthy  and  involve  thousands  of  numbers  in  the  case  of  these  scenarios,  manipulating  the 
data  was  cumbersome  and  occasionally  resulted  in  lost  data.  To  resolve  the  issue,  a  C 
program  was  written  and  used  as  a  filter  to  easily  gather  the  required  data  for  the  MOPs. 
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IV.  DATA  ANALYSIS 


This  chapter  presents  the  methodology  for  analyzing  the  results,  the  statistical 
tests  used  in  the  analysis,  and  the  results  from  the  analysis.  The  following  chapter 
provides  a  detailed  analysis  of  the  results  presented  in  this  chapter. 

A.  ANALYSIS  METHODOLOGY 

After  running  the  12  combinations  described  in  Figure  5,  with  10  replications  for 
each  cell,  summary  statistics  for  each  MOP  were  gathered.  The  raw  data  for  the 
experiment  is  presented  in  Appendices  A,  B,  and  C.  The  LER  is  the  total  red  entities 
killed  divided  by  the  total  blue  entities  killed.  Each  data  entry  for  detection  range  and  kill 
range  is  the  average  of  all  detection  ranges  (kill  ranges)  for  the  specified  mode  on  the 
given  run.  The  rounds  fired  data  is  the  total  of  all  direct  fire  rounds  shot  for  the  specified 
mode  on  the  given  run. 

The  data  was  then  analyzed  by  MOP.  First,  the  Analysis  of  Variance  (ANOVA) 
test  was  used  to  determine  if  differences  existed  between  the  means  of  the  Janus 
population  and  the  JLink  population  for  the  given  MOP.  ANOVA  determines  whether 
sample  means  from  different  populations  are  statistically  similar  by  analyzing  the  ratio  of 
the  variance  estimates.  ANOVA  also  helps  identify  interactions  occurring  between  the 
three  factors  of  the  experiment  -  scenario  type,  environment,  and  mode  (Janus/JLink)  - 
with  respect  to  the  given  MOP. 

The  ANOVA  test  identifies  whether  their  are  differences  in  the  means  between 
the  two  populations,  but  does  not  indicate  where  the  difference  occurred.  To  identify 
where  the  differences  in  the  means  exist,  a  pair-wise  two  sample  /-test  was  applied  to 
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those  MOPs  whose  populations  were  deemed  statistically  different  by  the  ANOVA.  In 
this  case,  the  goal  was  to  determine  whether  the  results,  from  an  MOP  in  a  specific 
scenario  and  environment  executed  in  Janus  stand-alone,  are  statistically  similar  to  the 
results  of  the  same  MOP  under  the  same  conditions  executed  in  JLink.  Such  a  pair-wise 
comparison  highlights  where  differences  in  the  means  exist. 

For  this  thesis,  the  sample  size  is  relatively  small  and  the  population  variances  are 
unknown,  hence  a  two  sample  f-test  is  appropriate  in  this  case.  [Ref.  6:p.  357-359].  The 
assumptions  for  the  two  sample  /-test  are: 

1.  Both  samples  are  independent  random  samples  from  normal  populations. 

2.  The  two  populations  have  equal  variance. 

Data  sets  were  analyzed  to  determine  whether  assumptions  were  satisfied  prior  to 
applying  the  /-test. 

In  cases  where  the  assumptions  were  not  satisfied,  the  data  was  analyzed  in  the 
same  pair-wise  method  using  the  less  powerful  Mann-Whitney-Wilcoxon  (Wilcoxon) 
nonparametric  test.  The  Wilcoxon  test  does  not  require  that  the  data  satisfy  the  normality 
assumption  above.  The  Wilcoxon  test  was  also  performed  on  the  MOPs  which  did  satisfy 
the  assumptions  to  further  substantiate  the  results  of  the  /-test. 

The  data  analysis  concludes  with  a  discussion  of  the  variability  within  the  JLink 
system  to  determine  if  the  main  contributor  to  variance  comes  from  within  the  JLink 
system  or  from  within  the  simulation  runs. 
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B.  TOOLS  FOR  ANALYSIS 


1.  ANOVA 

The  ANOVA  determines  whether  sample  means  from  different  populations  are 
statistically  similar  and  identifies  interaction  between  factors.  The  design  of  the 
experiment  resulted  in  the  data  being  organized  into  a  3  x  2  x  2  fashion,  meaning  the 
experiment  tested  three  factors,  the  first  involving  three  levels,  and  the  second  and  third 
factors  involving  two  levels  (Figure  5).  The  first  factor,  scenario  type,  was  tested  at  three 
levels:  armored,  armored  coalition,  and  light  infantry.  The  second  factor,  environment, 
was  tested  at  two  levels:  SWA  and  HL.  The  third  factor,  mode,  was  tested  at  two  levels: 


Janus  stand-alone  and  JLink.  The  first  and  second  factors  were  blocking  factors,  while 


the  third  factor  acted  as  the  treatment  factor  which  this  thesis  intended  to  analyze. 


An  interpretation  of  the  ANOVA  table  follows.  Table  1  shows  the  results  of  the 


ANOVA  for  detection  ranges  by  the  blue  forces. 


Analysis  of  Variance  Table  for  Detection  Range 
Response:  MOE  (Detection  Range  Blue) 

Blue 

Terms  added 

sequentially  (first  to  last) 

Df  Sum  of  Sq  Mean  Sq  F  Value 

Pr  (F) 

Seen 

2 

95.34164 

47.67082 

6050.279 

0.0000000 

Env 

1 

2.51797 

2.51797 

319.576 

0.0000000 

Mode 

1 

4.1143:8 

4 .11438 

522.188: 

0.0000000 

Seen : Env 

2 

0.19275 

0.09638 

12.232 

0.0000163 

Seen : Mode 

2 

1.64627 

0.82313 

104.470 

0.0000000 

Env : Mode 

1 

0.09369 

0.09369 

11.892 

0.0008050 

Seen : Env: Mode 
Residuals 

2 

108 

0.00861 

0.85094 

0.00431 

0.00788 

0.547 

0.5805110 

Table  1.  ANOVA  Results  for  Blue  Detection  Ranges 


The  ANOVA  table  provides  results  for  analysis  of  blocking  factors,  scenario  type  and 


environment,  and  the  treatment  factor,  mode  (Janus/JLink).  The  focus  of  the  thesis  is  on 


the  treatment  factor,  mode,  to  assess  whether  Janus  and  JLink  produce  similar  results  for 
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given  MOPs.  The  highlighted  row,  labeled  Mode,  is  interpreted  as  follows.  The 
ANOVA  model  equation  for  Yyk ,  the  observation  recorded  for  the  /th  scenario,  and  yth 

environment,  and  kth  mode,  is 

Yijk  =  M  +  A  +  Tj  +Yk+  £ikj 

where  jil  is  the  overall  mean 

A  is  the  effect  of  the  /th  treatment 

Tj  is  the  effect  of  the  j'th  environment 

yk  is  the  effect  of  the  kth  mode 

£ijk  is  the  experimental  error  ~  Normal  (0,cr2), 

and  it  is  initially  assumed  that  there  is  no  interaction  between  factors  [Ref.  7:  p.  532].  In 
words,  the  recorded  observations  for  Blue  Detection  Ranges,  separated  by  mode, 
(highlighted  row  above)  are  interpreted  as  follows: 

Janus  Blue  Det  Rng  =  overall  mean  Blue  Det  Rng  +  ScenEffect  +  EnvEffect  +  JanusModeEffect  +  error. 
Similarly,  every  JLink  recorded  observation  is  interpreted  as  follows: 

JLink  Blue  Det  Rng  =  overall  mean  Blue  Det  Rng  +  ScenEffect  +  EnvEffect  +  JLinkModeEffect  +  error. 
Assuming  the  model  is  additive,  subtracting  the  JLink  equation  from  the  Janus  equation 

returns  a  model  which  isolates  the  MOP  in  terms  of  the  mode  effects  (and  error): 

Janus  MOP  -  JLink  MOP  =  Janus  Mode  Effect  -  JLink  Mode  Effect  +  error. 

In  this  case,  the  zero  p-value  in  the  Mode  row  means  that  the  null  hypothesis 

Hg :  mean  Janus  Blue  Det  Rng  =  mean  JLink  Blue  Det  Rng 
is  strongly  rejected.  The  ANOVA  does  not,  however,  identify  where  the  difference 
occurred. 
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Figure  6  shows  interaction  plots  for  scenario: environment,  scenario: mode,  and 
environment: mode  for  the  same  MOP,  blue  detection  ranges.  The  interpretation  of  the 
scemmode  interaction  is  as  follows.  The  null  hypothesis  tested  is 

AvgJanusBlueDetRng  —  AvgJLinkBlueDetRng  =  A,  seen#] 

H0 :  AvgJanusBlueDetRng  -  AvgJLinkBlueDetRng  =  A,  seen# 2  >AllA  s  the  same. 

AvgJanusBlueDetRng  —  AvgJLinkBlueDetRng  =  A,scen#3 

In  this  case,  the  p-value  for  scemmode  interaction  is  zero,  thus  rejecting  the  null 
hypothesis.  This  result  can  be  interpreted  as  saying,  not  only  are  the  sample  means  for 
blue  detection  ranges  different  between  Janus  and  JLink,  the  difference  varies  between 
scenario.  Figure  6  highlights  with  arrows  where  the  difference  in  means  are  considerably 
larger  between  Janus  and  JLink  for  scenario  #1  compared  to  the  difference  in  scenario  #3. 


Figure  6.  Interaction  Plots  for  Blue  Detection  Ranges 
The  value  of  the  interaction  plots  for  this  thesis  is  that  they  provide  insights  as  to 
the  cause  of  disparate  results  between  Janus  and  JLink.  For  instance,  the  scemmode 
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interaction  plot  shows  that  greater  differences  in  blue  detection  ranges  occur  in  armored 
and  armored  coalition  scenarios  as  compared  to  light  infantry  scenarios.  This  finding 
indicates  that  situations  peculiar  to  armor  and  armor  coalition  scenarios  contribute  most 
to  the  gap  in  blue  detection  ranges  between  Janus  and  JLink. 

2.  Two  Sample  /-Test 
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rejection  region  of  the  null  hypothesis,  which  would  be  otherwise  rejected  5%  of  the  time 
simply  by  chance  if  the  original  a  was  used.  When  the  f-test  was  applied  to  an  MOP,  six 
different  pair-wise  tests  were  necessary  to  encompass  three  scenarios  executed  in  two 
environments.  Since  a  total  of  6  comparisons  were  conducted,  the  new  confidence  level 
used  for  these  /-tests  was 

anew=anld  /  6  =  0.0083. 

b.  Methods  for  Testing  Assumptions 

Before  applying  the  two  sample  /-tests,  the  assumptions  of  normality  and 
common  variance  between  sample  populations  were  considered.  Two  tests  were 
conducted  to  determine  if  the  assumption  of  normality  was  satisfied.  First,  Quantile- 
Quantile  Plots  (Q-Q  Plots)  for  the  normal  distribution  were  plotted.  The  results  are 
shown  in  Appendix  F.  The  population  is  considered  normally  distributed  if  the  plot 
produces  a  generally  straight  line. 

The  Kolmogorov-Smirnov  (KS)  Goodness  of  Fit  (GOF)  was  also  used  to 
test  for  the  normality  assumption.  The  test  determines  if  a  sample  population  follows  a 
prescribed  distribution.  The  KS  GOF  tests  the  following  hypothesis: 

Hg:  Sample  Population  ~  Normal  (x,s2) ,  versus 
Ha  :  True  cdf  does  not  equal  normal  distribution  for  at  least  one  sample  point. 

For  small  sample  sizes,  such  as  10,  the  KS  GOF  rarely  rejects  the  null  hypothesis  at  a 
a=0.05  level  of  significance.  Because  of  this  fact,  the  KS  GOF  was  used  as  a  secondary 
method  to  test  for  normality  after  visually  inspecting  the  Q-Q  Plots.  The  results  of  the  KS 
GOF  are  found  in  Appendix  G,  with  highlighted  areas  being  those  MOPs  with  p-values 
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less  than  0.05,  which  rejects  the  null  hypothesis.  The  p-value  is  the  smallest  level  of 
significance  at  which  Ha  would  be  rejected  when  a  specified  test  procedure  is  used  on  a 
given  data  set  [Ref.  6:p.  334].  In  general,  the  results  of  the  test  concur  with  the  findings 
from  the  Q-Q  Plots. 

The  next  assumption  required  for  the  two  sample  /-test  is  that  both 
samples  have  a  common  variance.  The  test  used  to  check  the  variance  assumption  was 
the  F-test  for  common  variances.  The  test  determines  if  the  ratio  of  the  variances  is  equal 
to  one.  In  this  case,  a  two-sided  F-test  is  used  with  a  oc=0.05  level  of  significance.  The 
hypothesis  tested  is 


jj  (J  JanusMOP  1 

H  o' - 2 - = 

(J  JLinkMOP 


rr  (7  JanusMOP 

versus ,  Ha  : — - - 

(7  JLinkMOP 


*1. 


The  results  of  the  F-test  are  in  Appendix  H,  with  the  highlighted  portions  being  those 
MOPs  which  rejected  the  null  hypothesis  at  the  a=0.05  level  of  significance  and  did  not 
satisfy  the  common  variance  assumption. 

For  MOPs  detection  range,  kill  range,  and  rounds  fired,  the  /-test  was 
applied  regardless  of  the  status  of  the  assumptions  in  order  to  provide  confidence 
intervals  for  further  analysis.  The  resulting  confidence  intervals  and  p-values  are  found  in 
Appendix  I. 
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c. 


Power  of  the  Two  Sample  f-Test 


Power  is  defined  as  the  probability  of  rejecting  a  false  null  hypothesis 
[Ref.  8:p.  79].  The  power  of  the  t- test  is  the  probability  of  detecting  an  acceptable 
difference  A,  in  the  sample  means  for  each  MOP.  In  terms  of  the  null  and  alternative 
hypothesis,  power  is  the  probability  of  rejecting 

Ho:A  =  0, 

in  favor  of 


Ha  :  A  =  predetermined  value, 


for  any  true  Ha . 

The  power  of  the  two-sample  r-test  is  a  function  of  the  sample  sizes  n  and 
m,  the  significance  level  a,  the  population  standard  deviation  <7,  and  the  real  difference 
between  the  sample  means  A  [Ref.  7  :p.  354,  394].  In  this  study,  all  scenarios  were  run  an 
equal  number  of  times;  therefore  the  sample  sizes  were  n  =  m  =  10  for  each  MOP.  The 
chosen  significance  level  is  a  =  0.05.  Using  the  assumption  that  the  two  populations 
from  which  the  MOPs  are  drawn  have  a  common  standard  deviation  c,  the  pooled  sample 
standard  deviation  is  used  to  estimate  o  as  follows: 


Determining  the  real  A,  or  the  value  of  the  difference  between  means  that 
is  considered  significant,  is  rather  subjective.  For  FASCAM  kills  and  chemical  kills,  a 
10%  difference  would  generally  be  acceptable  between  the  Janus  stand-alone  and  JLink 
results.  A  total  of  10  units  were  sent  through  the  FASCAM  minefields  for  all  scenarios. 
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resulting  in  a  value  of  A  =  1  for  FASCAM.  A  total  of  20  units  were  sent  through  the 
chemical  artillery  for  scenarios  1  and  2,  and  200  units  (due  to  aggregation  of  infantry) 
were  sent  through  chemical  artillery  for  scenario  3.  This  results  in  a  value  of  A  =  2  for 
chemical  artillery  in  scenarios  1  and  2,  and  A  =  20  for  scenario  3.  The  accepted 
difference  in  mean  LERs  is  0.3.  This  difference  is  based  on  the  expected  attrition  of  each 
scenario  and  the  impact  such  a  difference  would  make  on  the  outcome  of  the  battle.  The 
accepted  difference  in  the  mean  rounds  fired  is  approximately  20%  of  the  total  Janus 
rounds  fired  for  each  scenario. 

The  accepted  difference  in  mean  detection  ranges  and  kill  ranges  is  5%  of 
the  predominant  weapon’s  capabilities.  The  predominant  weapon  is  defined  as  the  most 
widely  used  weapon  in  the  scenario,  in  this  case  the  tank  for  scenarios  1  and  2,  and  the 
rifle  for  scenario  3.  The  maximum  detection  range  for  the  tank  and  rifle  is  six  and  two 
kilometers  respectively,  resulting  in  a  A  =  0.3  for  scenarios  1  and  2,  and  a  A  =  0. 1  for 
scenario  3.  The  maximum  kill  range  for  the  tank  is  four  kilometers,  while  the  rifle  has  a 
kill  range  of  one  kilometer.  This  results  in  a  A  =  0.2  for  killing  ranges  in  scenarios  1  and 
2,  and  A  =  0.05  for  scenario  3.  The  rationale  for  selecting  5%  for  the  killing  range 
follows.  Assuming  the  tank  travels  at  20  kilometers  per  hour,  then  a  5%  difference  in 
killing  range  results  in  an  additional  killing  range  of  0.2  kilometers,  or  additional  standoff 
time  of  0.6  minutes.  Since  a  tank  is  able  to  fire  four  rounds  per  minute,  an  acceptable 
difference  of  two  additional  rounds  will  be  fired  by  either  the  Janus  tank  or  JLink  tank. 

The  next  step  in  determining  the  power  is  to  calculate  <j),  the  non-centrality 
parameter  as  follows: 
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where  <7-  is  the  standard  deviation  of  the  sample  estimator  for  A  [Ref.  7:  p.  355,  394]. 
Further, 


This  equation  illustrates  that  as  sample  size  increases,  (p  also  increases,  and  likewise  the 
power  of  the  test. 

Appendix  D  shows  the  values  required  to  calculate  power.  Table  2  shows 
the  calculated  power  values  for  each  MOP  by  scenario  and  environment.  The  power  is 
determined  by  entering  the  noncentral  T  chart  at  the  appropriate  Rvalue,  using  18  degrees 
of  freedom  (since  two  means  were  estimated  from  20  total  data  points),  and  reading  the 
corresponding  power  value  [Ref.  7:p.  355].  So,  for  example,  our  sample  of  size  10  ought 
to  detect  a  20%  difference  in  “rounds  fired”  about  54%  of  the  time  (Table  2,  Scenario  #1, 
SWA).  The  results  in  Table  2  show  that  the  experiment  produces  power  values  greater 
than  50%  of  detecting  a  difference  A  in  the  sample  means  for  most  MOPs.  Also,  power 
values  for  detection  ranges  and  kill  ranges  are  approximately  70%  or  greater  in  most 
scenarios.  Power  values  in  the  ranges  noted  above  are  generally  sufficient  for  analysis  of 


combat  simulations. 


Scenario#!,  SWA 

FASCAM 

Chem  Kills 

LER 

DetRng 

Kill  Rng 

Rnds 

Power 

1.00 

1.00 

.72 

1.00 

.88 

w 

Scenario  #1,HL 

FASCAM 

Chem  Kills 

LER 

DetRng 

Kill  Rng 

Rnds 

Power 

.31 

.81 

.54 

1.00 

.93 

.29 

•  Scenario  #2,  SWA 

FASCAM 

Chem  Kills 

LER 

DetRng 

Kill  Rng 

Rnds 

Power 

.28 

1.00 

.83 

1.00 

.56 

.25 

Scenario  #2,  HL 

FASCAM 

Chem  Kills 

LER 

DetRng 

Kill  Rng 

Rnds 

Power 

.40 

.88 

.62 

1.00 

.94 

.40 

Scenario  #3,  SWA 

FASCAM 

Chem  Kills 

LER 

DetRng 

Kill  Rng 

Rnds 

Power 

.40 

1.00 

.40 

1.00 

.69 

1.00 

Scenario  #3,  HL 

FASCAM 

Chem  Kills 

LER 

DetRng 

Kill  Rng 

Rnds 

Power 

.72 

1.00 

<20 

1.00 

.31 

.986 

Table  2.  Power  Values  for  MOPs 


3.  Wilcoxon  Test 

To  substantiate  the  results  of  the  two  sample  Mest  and  to  test  those  MOPs  which 
did  not  satisfy  the  Mest  assumptions,  the  less  powerful  Mann-Whitney-Wilcoxon 
(Wilcoxon)  nonparametric  test  was  applied  to  compare  the  sample  means.  The  Wilcoxon 
test  uses  the  same  two-tailed  hypothesis  test  as  the  two  sample  Mest.  The  nonparametric 
test  does  not  require  that  the  data  satisfy  assumptions  of  normality  and  common  variance. 
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The  Wilcoxon  test  begins  by  rank-ordering  all  samples  from  both  populations  and 

assigning  ranks  1  through  m+n  (20  in  this  case)  to  all  values.  If  sample  values  are  exactly 

equal,  each  value  is  assigned  the  average  of  the  ranks  that  would  have  been  assigned  had 

there  been  no  tie.  The  test  statistic  is  found  by  first  finding  the  sum  (S)  of  the  ranks 

assigned  to  the  observations  from  the  first  population.  The  test  statistic  is  given  by 

T_s  n(n  +  l) 

2 

The  test  statistic  is  then  compared  to  the  critical  value  for  a  two-sided  test.  The  null 
hypothesis  is  rejected  if 

T<coa/ 2  or  T  >  ft),_a/2 , 

where  ft)a/2is  the  a/ 2 quantile  of  T  [Ref.  9:p.  224-226].  Conover  [Ref.  9:  Table  8] 

provides  a  table  of  critical  values  for  various  quantiles  of  T.  The  confidence  level  a  is 
determined  using  the  same  Bonferroni  method  applied  in  the  /-test.  As  with  the  /-test,  the 
confidence  level  used  was 

aold  /  6  =  0.0083 . 

C.  ANALYSIS  OF  MOPs 

The  data  set  used  to  analyze  FASCAM  kills,  chemical  kills,  and  LERs  is  found  in 
Appendix  A.  The  MOPs  detection  range,  kill  range,  and  rounds  fired  were  analyzed  by 
side  (Appendix  B)  to  provide  a  clearer  interpretation  of  the  results  and  additional 
information  to  whether  JLink  is  accurately  simulating  units  performing  a  specific  type  of 
mission,  in  this  case  attacking  and  defending. 
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As  previously  mentioned,  the  methodology  of  analyzing  the  MOPs  began  with  the 
ANOVA  test.  The  ANOVA  considers  all  three  factors  in  the  experiment  at  all  levels  to 
determine  if  the  sample  mean  of  a  particular  MOP  in  Janus  is  statistically  similar  to  the 
sample  mean  of  the  same  MOP  in  JLink.  The  results  of  the  ANOVA  tests  in  Appendix  E 
conclude  that  significant  differences  exist  in  the  sample  means  between  the  two  modes, 
except  in  the  instance  of  red  rounds  fired.  These  results  indicate  that  the  multiple  factors 
in  the  experiment  cause  sample  means  to  be  different  when  analyzed  as  a  whole.  The 
reasons  for  the  differences  are  initially  apparent  after  inspecting  the  raw  data  in 
Appendices  A  and  B.  Specifically,  Janus  and  JLink  produced  significantly  different 
chemical  kills  for  every  scenario,  and  detection  ranges  and  kill  ranges  are  consistently 
larger  for  JLink.  Given  these  disparate  inputs  to  the  ANOVA  test,  one  would  expect  the 
test  to  generate  statistically  different  sample  means  between  the  two  modes.  To  provide 
further  insight  to  the  particular  differences  between  Janus  and  its  distributed  counterpart, 
the  analysis  proceeded  by  identifying  significant  interactions  between  factors  for  each 
MOP  and  applying  the  pair-wise  two  sample  /-test  and  Wilcoxon  test  to  determine  where 
differences  existed,  leading  to  recommendations  for  improvements  to  JLink. 

1.  FASCAM 

FASCAM  was  tested  using  the  results  in  Table  3. 


44 


Summary  Statistics 

Janus  Stand- 

JLink  vs 

Alone 

JLink 

Armored  SWA 

Average 

2.7 

■■ 

Variance 

2.68 

Stand  Dev 

1.63 

Armored  HL 

Average 

2.9 

1.5 

Variance 

2.99 

.72 

Stand  Dev 

1.73 

.85 

Armored  Coalition  SWA 

Average 

2.3 

Variance 

2.68 

Stand  Dev 

1.63 

Armored  Coalition  HL 

Average 

2.6 

.6 

Variance 

2.49 

.27 

Stand  Dev 

1.58 

.52 

Light  Infantry  SWA 

Average 

Variance 

— 

.2 

.4 

Stand  Dev 

.63 

Light  Infantry  HL 

Average 

Variance 

1 

1.11 

.6 

.27 

Stand  Dev 

1.05 

.52 

Table  3.  FASCAM  Results 


First,  the  t-test  assumptions  were  examined.  Table  4  summarizes  the  results  of 
visually  inspecting  the  Q-Q  Plots  in  Appendix  F  and  the  results  of  the  KS  GOF  test  in 
Appendix  G.  The  highlighted  portions  are  those  areas  which  did  not  satisfy  the  normal 
assumption. 


Scenario#!,  SWA 

Scenario  #1,HL 

Scenario  #2,  SWA 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Normal 

yes 

yes 

yes 

yes 

yes 

yes 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Normal 

rib 

no 

yes 

no 

yes 

(ill 

Table  4.  FASCAM  Normality 


Next,  the  F-test  was  used  to  check  for  common  variance  between  the  Janus  and 
JLink  populations.  The  results  of  the  F-tests  are  in  Appendix  H.  Table  5  summarizes  the 
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results  of  the  tests  for  normality  and  common  variance  for  each  scenario.  The  highlighted 
portions  are  those  populations  which  failed  either  the  normality  assumption  or  the 
common  variance  assumption,  or  both.  The  two  sample  /-test  was  only  performed  on 
those  Janus/JLink  pairs  that  met  the  normality  assumption  and  passed  the  common 
variance  test. 


Scenario#!,  SWA 

Scenario  #1,HL 

Scenario  #2,  SWA 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Assumptions 

)£S 

HI 

yes 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Assumptions 

Hi 

Hi 

Table  5.  FASCAM  Satisfying  Normality  and  Common  V  ariance 
Appendix  I  provides  results  of  the  pair-wise  two  sample  /-tests,  to  include  p- 
values  and  confidence  intervals  (Cl),  and  highlights  those  pairs  which  reject  the  null 
hypothesis.  Table  6  summarizes  the  FASCAM  results.  Note  that  both  pairs  tested  failed 
to  reject  the  null  hypothesis. 


Scenario#!,  SWA 

Scenario  #1,HL 

Scenario  #2,  SWA 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

fail  to  nej  null  hyp 

na 

fail  to  rej  null  hyp 

i  \  ;  .  •  :  :•  •  • 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Janus 

_ 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

na 

na 

na 

Table  6.  FASCAM  Results  of  Two  Sample  /-Test 
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The  Wilcoxon  test  was  then  applied  to  all  pair-wise  comparisons  of 


FASCAM.  Appendix  J  provides  the  results  of  the  Wilcoxon  test,  and  highlights  those 
pairs  which  reject  the  null  hypothesis.  Tables  7  summarizes  the  FASCAM  results  and 
highlights  those  pairs  which  reject  the  null  hypothesis  that  the  two  sample  averages  are 
statistically  similar. 


Scenario#!,  SWA 

Scenario  #1,  HL 

Scenario  #2,  SWA 

Mode 

Janus 

JUnk 

Janus 

JUnk 

Janus 

JLink 

Results 

fail  to  iej  null  hyp 

fail  to  mj  null  hyp 

fail  to  rej  null  hyp 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

i 

Janus 

JLink 

Janus 

JUnk 

Janus 

JUnk 

Results 

fail  to  rej  null  hyp 

fail  to  rej  null  hyp 

Table  7.  FASCAM  Results  of  Wilcoxon  Test 


2.  Chemical  Kills 

The  chemical  kills  MOP  was  analyzed  using  the  results  in  Table  8. 
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Summary  Statistics 

Janus  Stand- 
Alone 

JLink  vs 
JLink 

Armored  SWA 

Average 

4.4 

.3 

Variance 

.49 

.23 

Stand  Dev 

.70 

.48 

Armored  HL 

Average 

12.3 

2.5 

Variance 

3.12 

1.39 

Stand  Dev 

1.77 

1.18 

Armored  Coalition  SWA 

Average 

3.8 

.1 

Variance 

.4 

.1 

Stand  Dev 

.63 

.32 

Armored  Coalition  HL 

Average 

12.3 

2.9 

Variance 

2.68 

.99 

Stand  Dev 

1.64 

.99 

Light  Infantry  SWA 

Average 

Variance 

9.9 

10.1 

.6 

1.6 

Stand  Dev 

3.18 

1.26 

Light  Infantry  HL 

Average 

Variance 

69.6 

1.6 

36.6 

77.6 

Stand  Dev 

1.26 

8.81 

Table  8.  Chemical  Results 


The  analysis  applied  the  two  sample  /-test  to  identify  the  specific  scenarios  which 
produced  statistically  different  chemical  kills  between  Janus  and  JLink.  First,  the 
assumptions  were  checked  in  the  same  manner  as  with  FASCAM.  Normality  was 
determined  based  on  the  visual  inspection  of  the  Q-Q  Plots  and  the  results  of  the  KS  GOF 
test.  Then,  common  variance  was  determined  based  on  the  results  of  the  F-test.  Table  9 
summarizes  the  results  of  the  tests  for  normality  and  common  variance  for  each  scenario. 
The  highlighted  portions  are  those  populations  which  failed  either  the  normality 
assumption  or  the  common  variance  assumption,  or  both.  Again,  the  two  sample  /-test 
was  only  performed  on  those  Janus/JLink  pairs  that  met  the  normality  assumption  and 
passed  the  common  variance  test. 
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Table  9.  Chemical  Kills  Satisfying  Normality  and  Common  Variance 

The  r-test  results  in  Appendix  I  show  that  the  one  pair-wise  comparison  tested 
rejected  the  null  hypothesis  that  the  average  number  of  chemical  kills  between  Janus  and 
JLink  are  statistically  similar  in  scenario  #1,  HL.  The  result  is  summarized  (for 
consistency)  in  Table  10. 


Scenario#!,  SWA 

Scenario  #1,HL 

Scenario  #2,  SWA 

Janus 

JLink 

Janas 

JLink 

Janus 

JLink 

na 

nejnull 

ill 

na 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  III. 

Janus 

JLink 

Janus 

JLink 

Janas 

JLink 

na 

na 

na 

Table  10.  Chemical  Results  of  Two  Sample  /-Test 


The  Wilcoxon  test  was  then  applied  to  all  pair-wise  comparisons  of  chemical 
kills.  Appendix  J  provides  the  results  of  the  Wilcoxon  test,  and  highlights  those  pairs 
which  reject  the  null  hypothesis.  Tables  1 1  summarizes  the  chemical  results  and 
highlight  those  pairs  which  reject  the  null  hypothesis  that  the  two  sample  averages  are 
statistically  similar. 
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Scenario  #1,  SWA 

Scenario  #1,HL 

Scenario  #2,  SWA 

Mode 

Janus 

Jlink 

Janus 

Jlink 

Janus 

JLink 

Results 

lejnuBhy^11 

^nuUhyp 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

Jlink 

Results 

iirejtplhyp 

ignuflftyp1 

Table  11.  Chemical  Results  of  Wilcoxon  Test 


3.  Detection  Range  (Blue/Red) 

The  detection  range  MOP  was  analyzed  using  the  results  in  Tables  12  and  13.  As 


previously  mentioned,  the  data  for  detection  range  was  analyzed  separately  by  blue  and 
red  force. 


Summary  Statistics 

Janus  Stand- 

JLinkvs 

Alone 

JLink 

Armored  SWA 

Average 

3.689 

' 

4.171 

Variance 

.010 

.005 

Stand  Dev 

.098 

.072 

Armored  HL 

Average 

3.330 

3.951 

Variance 

.005 

.012 

Stand  Dev 

.072 

.107 

Armored  Coalition  SWA 

Average 

3.619 

4.073 

Variance 

.033 

.011 

Stand  Dev 

.181 

.104 

Armored  Coalition  HL 

Average 

3.362 

3.948 

Variance 

.009 

.006 

Stand  Dev 

.096 

.074 

Light  Infantry  SWA 

Average 

Variance 

2.067 

.002 

2.075 

.001 

Stand  Dev 

.040 

.027 

Light  Infantry  HL 

Average 

Variance 

1.648 

.001 

1.719 

.001 

Stand  Dev 

.033 

.036 

Table  12.  Blue  Detection  Range  Results 
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Summary  Statistics 

Janus  Stand- 
Alone 

JLink  vs 
JLink 

Armored  SWA 

Average 

Variance 

3.389 

.021 

Stand  Dev 

HH 

.146 

Armored  HL 

Average 

Variance 

2.882 

.034 

Stand  Dev 

.183 

Armored  Coalition  SWA 

Average 

Variance 

2.212 

.011 

3.242 

.027 

Stand  Dev 

.103 

.163 

Armored  Coalition  HL 

Average 

Variance 

1.888 

.007 

2.624 

.054 

Stand  Dev 

.081 

.232 

Light  Infantry  SWA 

Average 

Variance 

1.791 

.001 

2.008 

.001 

i 

Stand  Dev 

.030 

.030 

Light  Infantry  HL 

Average 

Variance 

1.559 

.001 

1.672 

.002 

Stand  Dev 

.027 

.041 

Table  13.  Red  Detection  Range  Results 


The  ANOVA  interaction  plots  in  Appendix  E  for  both  blue  and  red  detections  show 
significant  interaction  between  the  scenario  and  the  mode,  illustrated  by  the  larger  gap  in 
detection  differences  for  scenarios  1  and  2  as  compared  to  the  detection  difference  for 
scenario  3.  Such  interaction  is  expected  given  that  scenarios  1  and  2  used  weapon 
systems  with  detection  capability  up  to  six  kilometers,  whereas  scenario  3  used  infantry 
units  with  detection  capability  of  roughly  two  kilometers. 

Testing  for  r-test  assumptions  showed  that  all  detection  populations  satisfied  the 
normality  assumption  and  only  the  red  detection  range  in  scenario  #  2,  HL  rejected  the 
assumption  of  common  variance.  Tables  14  and  15  show  the  summarized  results  of  the  t- 
test  for  blue  and  red  detection  ranges,  respectively. 
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Scenario#!,  SWA 

Scenario  #1,HL 

Scenario  #2,  SWA 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

ngnullh>p 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

JUnk 

Janus 

JLink 

Janus 

JLink 

Results 

fail  to  lej  null  hyp 

Table  14.  Blue  Detection  Range  Results  of  t-Test 


Scenario#!,  SWA 

Scenario  #1,  HL 

Scenario  #2,  SWA 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JUnk 

Results 

tejnuiihyp 

liej  riutfcy£ ' 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

JUnk 

Janus 

JUnk 

Janus 

JLink 

Results 

na 

i^jnullh>p 

p^tMhypi 

Table  15.  Red  Detection  Range  Results  of  /-Test 


The  Wilcoxon  test  was  also  applied  to  all  pair-wise  comparisons  of  detection 
ranges.  Tables  16  and  17  summarize  the  results  from  Appendix  J  and  highlight  those 
pairs  which  reject  the  null  hypothesis  that  the  two  sample  averages  are  statistically 
similar. 
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Scenario#!,  SWA 

Scenario#!,  HL 

Scenario  #2,  SWA 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

ighidlhjp' 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

null  hyp 

fail  to  lej  null  hyp 

Table  16.  Blue  Detection  Range  Results  of  Wilcoxon  Test 


Scenario#!,  SWA 

Scenario  #1,HL 

Scenario  #2,  SWA 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

iHlisii 

0-  . 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

^  nidi  hyp 

Table  17.  Red  Detection  Range  Results  of  Wilcoxon  Test 


4.  Kill  Ranges  (Blue/Red) 

The  kill  range  MOP  was  analyzed  using  the  data  in  Tables  19  and  20.  The  data 
for  kill  ranges  was  analyzed  separately  by  blue  and  red  force. 
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Armored  SWA 


Armored  HL 


Armored  Coalition  SWA 


Armored  Coalition  HL 


Light  Infantry  SWA 


Light  Infantry  HL 


Summary  Statistics 

Janus  Stand- 

JLink  vs 

Alone 

JLink 

Average 

2.074 

2.394 

Variance 

.123 

.006 

Stand  Dev 

.350 

.076 

Average 

1.955 

2.556 

Variance 

.045 

.061 

Stand  Dev 

.212 

.246 

Average 

1.987 

2.325 

Variance 

.136 

.216 

Stand  Dev 

.369 

.465 

Average 

1.831 

2.145 

Variance 

.050 

.120 

Stand  Dev 

.224 

.346 

Average 

1.477 

1.626 

Variance 

.005 

.001 

Stand  Dev 

.072 

.027 

Average 

1.301 

1.375 

Variance 

.003 

.001 

Stand  Dev 

.023 

Table  18.  Blue  Kill  Range  Results 


Armored  Coalition  SWA 


Summary  Statistics 

Janus  Stand- 
Alone 

Average 

2.847 

Variance 

.033 

Stand  Dev 

.181 

Average 

2.607 

Variance 

.016 

Stand  Dev 

.127 

Average 

3.00 

Variance 

.025 

Stand  Dev 

.158 

Average 

2.626 

Variance 

.018 

Stand  Dev 

.135 

Average 

1.313 

Variance 

.005 

Stand  Dev 

.067 

Average 

1.317 

Variance 

.004 

Stand  Dev 

.067 

JLink  vs 
JLink 


3.100 

.061 

.246 


Armored  Coalition  HL 


Light  Infantry  SWA 


Light  Infantry  HL 


Table  19.  Red  Kill  Range  Results 

The  ANOVA  interaction  plots  illustrate  a  significant  interaction  between  the  environment 
and  the  mode  for  red  kill  ranges.  This  interaction  is  interpreted  as,  not  only  are  the  mean 
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red  kill  ranges  different  between  Janus  and  JLink,  the  differences  are  greater  in  HL  terrain 
than  SWA  terrain. 

Testing  for  /-test  assumptions  showed  that  all  red  kill  range  populations  satisfied 
both  the  normality  and  common  variance  assumptions.  Table  20  summarizes  which 
scenarios  satisfied  the  assumptions  for  blue  kill  ranges. 


Scenario#!,  SWA 

Scenario  #1,HL 

Scenario  #2,  SWA 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Assumptions 

1®. 

yes 

pH 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Assumptions 

yes 

m 

Table  20.  Blue  Kill  Ranges  Satisfying  Normality  and  Common  Variance 

Tables  21  and  22  summarize  the  results  from  Appendix  I  of  the  two  sample  /-test 
for  blue  and  red  kill  ranges,  respectively. 


Scenario#!,  SWA 

Scenario  #1,HL 

Scenario  #2,  SWA 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

na 

na 

t  :  f3  ' 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

fail  to  rej  null  hyp 

na 

na 

Table  21.  Blue  Kill  Range  Results  of  /-Test 
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Scenario#!,  SWA 

Scenario  #1,HL 

Scenario  #2,  SWA 

Mode 

Janus 

Jlink 

Janus 

Jlink 

Janus 

Jlink 

Results 

fail  to  rej  null  hyp 

^raillhjp 

fail  to  rej  null  hyp 

?■ ?•' '  •  :  ■■ . ;  •  ■  ;  - . y;  ■ . $  jf ; .  ■: } 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

Jlink 

Janus 

_ 

Jlink 

Janus 

Jlink 

Results 

fail  to  rej  null  hyp 

fail  to  rej  null  hyp 

Table  22.  Red  Kill  Range  Results  of  t-Test 


The  Wilcoxon  test  was  then  applied  to  all  pair-wise  comparisons  of  kill  ranges. 
Tables  23  and  24  summarize  the  results  from  Appendix  J  and  highlight  those  pairs  which 
reject  the  null  hypothesis  that  the  two  sample  averages  are  statistically  similar. 


Scenario#!,  SWA 

Scenario#!,  HL 

Scenario  #2,  SWA 

Mode 

Janus 

Jlink 

Janus 

Jlink 

Janus 

Jlink 

Results 

fail  to  iej  null  hyp 

fail  to  iej  null  hyp 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

Jlink 

Janus 

Jlink 

Janus 

JLink 

Results 

fail  to  rej  null  hyp 

bsj  ttuEJiyp 

Table  23.  Blue  Kill  Range  Results  of  Wilcoxon  Test 
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Scenario#!,  SWA 

Scenario#!, HL 

Scenario  #2,  SWA 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

fail  to  rej  null  hyp 

fail  to  rej  null  hyp 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

fail  to  rej  null  hyp 

fail  to  rej  null  hyp 

Table  24.  Red  Kill  Range  Results  of  Wilcoxon  Test 


5.  Rounds  Fired  (Blue/Red) 


The  rounds  fired  MOP  was  analyzed  using  the  data  in  Tables  25  and  26.  The  data 


for  rounds  fired  was  analyzed  separately  by  blue  and  red  force. 


Summary  Statistics 

Janus  Stand- 

JLink  vs 

Alone 

JLink 

Armored  SWA 

Average 

85.3 

32.8 

Variance 

1292.23 

101.73 

Stand  Dev 

35.95 

10.09 

Armored  HL 

Average 

112.3 

58.1 

Variance 

1360.01 

40.32 

Stand  Dev 

36.88 

6.35 

Armored  Coalition  SWA 

Average 

56.5 

35.3 

Variance 

176.5 

206.01 

Stand  Dev 

13.29 

14.35 

Armored  Coalition  HL 

Average 

71.6 

73.6 

Variance 

424.04 

Stand  Dev 

20.59 

Light  Infantry  SWA 

Average 

Variance 

969.6 

63081.38 

621.3 

1896.46 

Stand  Dev 

251.16 

43.55 

Light  Infantry  HL 

Average 

Variance 

1347.6 

93197.38 

1341.5 

31205.83 

Stand  Dev 

305.28 

176.65 

Table  25.  Blue  Rounds  Fired  Results 


57 


Summary  Statistics 

Janus  Stand-  /.; 
Alone 

JLink  vs 
JLink 

Armored  SWA 

Average 

110.1 

195.3 

Variance 

1215.43 

983.79 

Stand  Dev 

34.86 

31.37 

Armored  HL 

Average 

531.9 

323.8 

Variance 

75729.21 

6535.07 

Stand  Dev 

275.19 

80.84 

Armored  Coalition  SWA 

Average 

144 

165.7 

Variance 

6945.78 

918.46 

Stand  Dev 

83.34 

30.31 

Armored  Coalition  HL 

Average 

523.4 

411.9 

Variance 

19137.16 

18999.88 

Stand  Dev 

138.34 

137.84 

Light  Infantry  SWA 

Average 

Variance 

1626 

11621.11 

— 

Stand  Dev 

107.80 

67.16 

Light  Infantry  HL 

Average 

Variance 

1365.5 

64852.5 

1594 

53604.44 

Stand  Dev 

254.66 

231.53 

Table  26.  Red  Rounds  Fired  Results 


The  results  of  the  ANOVA  test  in  Appendix  E  show  that  the  average  red  rounds  fired  in 
JLink  is  statistically  similar  to  the  average  red  rounds  fired  in  Janus,  as  evidenced  by  the 
p-value  of  .96  for  Mode.  Because  of  this  result,  no  further  statistical  tests  were  conducted 
on  the  red  rounds  fired.  Also,  the  results  of  the  ANOVA  interaction  plots  illustrate 
interaction  between  the  environment  and  mode  for  blue  rounds  fired.  The  difference 
between  Janus  and  JLink  in  the  mean  blue  rounds  fired  in  SWA  is  significantly  larger 
than  the  mean  blue  rounds  fired  in  HL. 

Next,  the  two  sample  t-test  was  applied  to  all  pairs  of  blue  rounds  fired.  Results 
from  testing  for  normality  and  common  variance  are  summarized  in  Table  27. 
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Scenario#!,  SWA 

Scenario  #1,  HL 

Scenario  #2,  SWA 

Mode 

Janus 

JUnk 

Janus 

JLink 

Janus 

JLink 

Assumptions 

B 

it 

R: 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Assumptions 

yes 

IB 

yes 

Table  27.  Blue  Rounds  Fired  Satisfying  Normality  and  Common  Variance 

Table  28  summarizes  the  results  from  Appendix  I  of  the  two  sample  t-test  for  blue 
rounds  fired. 


Scenario#!,  SWA 

Scenario  #1,  HL 

Scenario  #2,  SWA 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

na 

na 

na 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HI, 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

fail  to  rej  null  hyp 

_ 

na 

fail  to  rej  null  hyp 

Table  28.  Blue  Rounds  Fired  Results  of  1-Test 


The  Wilcoxon  test  was  then  applied  to  all  pair-wise  comparisons  of  blue  rounds 
fired.  Table  29  summarizes  the  results  from  Appendix  J  and  highlights  those  pairs  which 
reject  the  null  hypothesis  that  the  two  sample  averages  are  statistically  similar. 
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Scenario#!,  SWA 

Scenario  #1,HL 

Scenario  #2,  SWA 

Mode 

Janus 

JLink 

Janus 

JLink 

Janas 

JLink 

Results 

fail  to  rej  null  hyp 

:  •  '  .  '  '  =  •  .  ‘  ‘  ''  .  •  . 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

fail  to  rej  null  hyp 

i^nidlhyp 

fail  to  rej  null  hyp 

Table  29.  Blue  Rounds  Fired  Results  of  Wilcoxon  Test 


6.  Loss  Exchange  Ratio  (LER) 

The  LER  MOP  was  analyzed  using  the  results  in  Table  30. 


Summary  Statistics 

Janus  Stand- 

JLink  vs 

Alone 

JLink 

Armored  SWA 

Average 

2.182 

1.161 

Variance 

.072 

.074 

Stand  Dev 

.269 

.273 

Armored  HL 

Average 

1.659 

1.505 

Variance 

.156 

.049 

Stand  Dev 

.395 

.221 

Armored  Coalition  SWA 

Average 

1.377 

1.174 

Variance 

.070 

Stand  Dev 

.265 

Armored  Coalition  HL 

Average 

1.489 

1.835 

Variance 

.040 

.122 

Stand  Dev 

.201 

.350 

Light  Infantry  SWA 

Average 

Variance 

.937 

.011 

Stand  Dev 

wtBmm 

.104 

Light  Infantry  HL 

Average 

Variance 

2.36 

.436 

Stand  Dev 

1.002 

.660 

Table  30.  LER  Results 


The  analysis  applied  the  two  sample  Z-test  to  identify  the  specific  scenarios  which 


produced  statistically  different  LERs  between  Janus  and  JLink.  First,  the  t- test 


assumptions  were  checked  in  the  same  manner  as  before.  The  tests  revealed  that  all 
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scenarios  satisfied  both  the  normality  and  common  variance  assumption  except  scenario 
#3,  SWA.  The  results  of  the  Mests  are  summarized  in  Table  31. 


Scenario#!,  SWA 

Scenario  #1,  HL 

Scenario  #2,  SWA 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

fail  to  iej  null  hyp 

fail  to  rej  null  hyp 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

fail  to  rej  null  hyp 

na 

fail  to  rej  null  hyp 

Table  31.  LER  Results  of  Two  Sample  /-  l  est 


The  Wilcoxon  test  was  then  applied  to  all  pair-wise  comparisons  of  LERs.  Tables 
32  summarizes  the  LER  test  results  and  highlights  those  pairs  which  reject  the  null 
hypothesis  that  the  two  sample  averages  are  statistically  similar. 


Scenario#!,  SWA 

Scenario  #1,  HL 

Scenario  #2,  SWA 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

fail  to  rej  null  hyp 

failtoiej  null  hyp 

Scenario  #2,  HL 

Scenario  #3,  SWA 

Scenario  #3,  HL 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

Results 

failtoiej  null  hyp 

nmni! 

failtoiej  null  hyp 

Table  32.  LER  Results  of  Wilcoxon  Test 


D.  VARIABILITY  WITHIN  JLINK 

The  data  in  Appendix  C  are  the  results  of  10  JLink  runs,  using  the  same  scenarios 
and  environments  as  described  before,  except  that  the  same  random  number  seed  was 
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used  for  all  runs.  The  reason  for  conducting  these  runs  was  to  determine  the  variance 
within  the  JLink  system  and  compare  that  variance  to  runs  using  different  random  number 
seeds.  In  essence,  this  test  isolated  the  variance  contributed  by  JLink  alone. 

Inspecting  the  variances  in  Appendix  C,  and  comparing  them  to  the  variances  of 
the  runs  using  10  different  random  number  seeds  in  Appendix  A,  we  see  that  the  JLink 
variance  using  the  same  seed  was  generally  slightly  smaller  than  the  variance  in  Appendix 
A.  This  finding  indicates  that  most  of  the  variability  is  generated  within  the  runs,  instead 
of  from  the  changes  in  the  random  number  seed. 
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V.  ANALYSIS  OF  RESULTS 


The  results  of  the  statistical  tests  from  Chapter  IV  provide  a  firm  basis  for 
assessing  the  differences  between  the  JLink  system  and  its  parent  simulation,  Janus.  This 
chapter  will  analyze  the  results  of  the  ANOVA  test,  two  sample  r-test,  and  Wilcoxon  test 
and  offer  insights  to  the  causes  of  these  differences.  Since  the  ANOVA  test  rejected  the 
null  hypothesis  that  the  averages  were  statistically  similar  between  Janus  and  JLink  for  all 
MOPs  except  red  rounds  fired,  the  analysis  will  focus  primarily  on  where  the  differences 
occurred  based  on  the  results  of  the  r-test  and  Wilcoxon  test.  The  analysis  includes 
possible  causes  for  unexpected  results  and  provides  recommendations  to  resolve  those 
issues  derived  from  such  results.  The  analysis  of  the  results  is  presented  by  individual 
MOP. 

A.  FASCAM  KILLS 

The  results  from  the  two  sample  r-test  and  Wilcoxon  test  rejected  the  null 
hypothesis  that  the  average  FASCAM  kills  are  the  same  between  Janus  stand-alone  and 
JLink  in  only  one  instance,  for  scenario  #2,  HL.  This  would  lead  one  to  believe  that 
JLink  is  accurately  representing  FASCAM  under  various  conditions.  Also,  four  out  of  six 
times  Jlink  produced  average  results  which  were  within  the  accepted  tolerance  of  A  =  1 
kill.  However,  two  issues  must  be  addressed  before  drawing  a  final  conclusion  about 
JLink  FASCAM:  the  location  of  the  FASCAM  minefield  and  the  trend  in  the  data 
between  Janus  and  JLink. 
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As  mentioned  in  Chapter  III,  paragraph  C.  2,  the  player  fires  FASCAM  from  an 
artillery  platform  after  the  scenario  begins.  To  remove  aim-error  variability  due  to  human 
interaction,  all  runs  were  “pucked”  to  provide  a  common  firing  point  between  scenarios. 
Ballistic  errors  in  azimuth  and  elevation  are  another  source  of  location  variability  when 
firing  from  an  artillery  platform.  Janus  simulates  these  ballistic  errors  by  drawing  the  x 
and  y  coordinates  of  the  round’s  impact  point  from  a  normal  distribution.  So,  in  order  to 
ensure  that  a  FASCAM  minefield  is  placed  precisely  in  the  same  location  in  different 
runs,  the  runs  must  be  “pucked”  under  almost  identical  conditions  to  ensure  reasonable 
azimuth  and  elevation  errors.  The  experiment  satisfied  the  first  condition,  but  due  to 
JLink  architecture  and  different  random  number  generators,  dispersion  caused  by  the 
latter  condition  could  not  be  completely  controlled. 

To  reduce  the  dispersion  of  the  minefields,  FASCAM  was  fired  in  a 
relatively  small  geographic  location,  compared  to  the  entire  scenario.  By  firing  five 
minefields  per  run,  for  10  runs,  in  a  relatively  small  location,  one  would  expect  the  effect 
of  minefield  dispersion  to  be  negligible  over  the  entire  experiment.  However,  after 
replaying  some  of  the  scenarios  on  the  Janus  Analyst  Workstation  and  comparing  the 
locations  of  the  minefields  between  modes,  it  was  evident  that  minefield  dispersion 
would  contribute  to  dissimilar  FASCAM  kills  in  Janus  and  JLink.  Recall  that  blue  forces 
fired  FASCAM  at  specified  red  units  traveling  on  a  narrow  route.  Suppose  that  four  of 
the  five  FASCAM  minefields  fired  in  Janus  landed  on  the  designated  route  whereas  only 
three  JLink  minefields  landed  on  the  route,  due  to  ballistic  error.  Under  these 
circumstances,  the  experiment  would  not  isolating  the  effects  of  the  FASCAM 
minefields. 
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Another  issue  regarding  the  FASCAM  results  is  the  consistent  trend  towards 
greater  FASCAM  kills  in  Janus  as  compared  to  JLink.  Such  a  trend  forces  the  analyst  to 
proceed  with  caution  when  drawing  conclusions  from  the  statistical  tests.  Although  the 
tests  indicate  that  the  FASCAM  results  are  statistically  similar  between  modes  (except  in 
one  instance),  such  a  trend  may  indicate  that  the  JLink  FASCAM  does  not  produce  the 
same  number  of  kills  per  mine  as  Janus  FASCAM. 

Given  the  results  of  the  tests,  the  current  configuration  of  JLink  provides  a  fair 
representation  of  FASCAM  as  compared  to  Janus.  Although  there  appears  to  be  a  trend 
in  average  FASCAM  kills  favoring  Janus,  the  significance  of  the  trend  is  diminished 
somewhat  by  the  fact  that  one  out  of  every  six  runs  produced  greater  JLink  FASCAM 
kills  than  Janus.  The  trend  in  average  kills,  however,  does  merit  further  investigation. 
Additional  testing,  isolating  the  effects  of  FASCAM,  is  necessary  to  determine  if  the 
JLink  FASCAM  is  as  lethal  as  Janus  FASCAM. 

B.  CHEMICAL  KILLS 

In  all  cases,  the  t-test  and  the  Wilcoxon  test  rejected  the  null  hypothesis 
that  the  average  chemical  kills  are  the  same  between  Janus  stand-alone  and  JLink. 
Clearly,  JLink’ s  current  configuration  does  not  accurately  replicate  the  effects  of  Janus 
chemical  artillery. 

In  an  attempt  to  identify  the  cause  of  the  disparate  test  results  and  rectify 
the  shortcoming,  the  JLink  delivery  method  shown  in  Figure  2  was  investigated.  As 
aforementioned,  Janus  calculates  artillery  by  volley  as  opposed  to  individual  rounds,  as 
dictated  by  the  DIS  standard.  Calculating  the  artillery  by  volley  results  in  a  single  large 
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chemical  cloud  which  stays  together  for  a  much  longer  period  of  time  as  opposed  to 
individual  clouds  which  dissipate  quickly  and  cover  less  area.  Since  the  smaller  clouds 
dissipate  quickly,  entities  in  contact  are  affected  less  by  the  smaller  chemical  clouds  than 
by  the  larger  aggregated  cloud. 

In  order  to  correct  this  shortcoming,  the  World  Modeler  software  was  rewritten  so 
that  whenever  a  user  participates  in  a  JLink  to  JLink  game,  all  chemical  artillery  will  be 
fired  and  portrayed  by  volley  as  opposed  to  individual  rounds.  After  making  the 
corrections  in  the  code,  10  additional  runs  were  performed  using  the  SWA  armored 
scenario.  The  results  of  the  scenario  were  more  in  line  with  what  was  originally 
expected. 

The  ten  runs  of  both  Janus  and  JLink  yielded  an  average  of  10.3  and  10.5 
chemical  kills  respectively  (Table  33).  Because  the  data  satisfied  the  assumptions  of 
normality  and  common  variance  with  a  Q-Q  Plot  and  F-test  respectively,  a  two-sample  t- 
test  was  also  conducted.  Since  the  generated  test  statistic  0.7356  is  less  than  the  critical 
value  of  2.101,  the  conclusion  is  that  the  means  do  not  differ  significantly. 


RUN# 

Chemical  Kills 

Janus 

JLink 

1 

9 

9 

2 

11 

11 

3 

12 

11 

4 

9 

ii 

5 

9 

11 

6 

11 

12 

7 

11 

10 

8 

8 

11 

9 

11 

8 

10 

12 

11 

XBAR 

10.3 

10.5 

VAR 

2.011 

1.389 

STDE 

1.418 

1.179 

Table  33.  Chemical  Kills  for  Volley  Fire 
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These  results  support  the  finding  that  JLink  accurately  represents  chemical 
artillery  in  a  SWA  armored  scenario.  Additional  testing  using  other  scenarios  is 
necessary  to  conclude  that  JLink  accurately  portrays  chemical  artillery  under  varied 
conditions.  The  results  show  that  chemical  rounds  must  be  fired  in  volleys,  as  opposed  to 
individual  rounds,  whenever  a  user  plays  a  JLink  to  JLink  game.  Software  upgrades  will 
include  a  toggle  transparent  to  the  user  so  that  JLink  automatically  sends  out  volleys  as 
opposed  to  individual  rounds  whenever  a  user  indicates  that  the  other  participating  DIS 
simulation  is  another  JLink. 

C.  DETECTION  RANGES  (BLUE/RED) 

The  next  MOP  analyzed  was  detection  range,  analyzed  separately  by  blue  force 
and  red  force,  using  the  data  from  Appendix  B.  (The  headings  “Det  Ranges  Blue/Red”  in 
Appendix  B  refer  to  the  those  units  doing  the  detecting).  As  mentioned  previously, 
separating  the  detection  ranges  by  side  offers  a  clearer  interpretation  of  the  results  and 
additional  information  to  whether  JLink  is  accurately  simulating  units  performing  a 
specific  type  of  mission,  in  this  case  attacking  and  defending. 

The  two-sample  r-test  and  the  Wilcoxon  test  rejected  the  null  hypothesis  that  the 
average  blue  and  red  detection  ranges  are  similar  for  every  scenario  except  for  the  blue 
detection  ranges  in  scenario  #3,  SWA. 

Next,  the  thesis  compared  the  accepted  A  =  0.3  kilometer  tolerance  for  detection 
ranges  in  scenarios  1  and  2,  and  A  =  0.1  kilometers  for  scenario  3,  to  the  confidence 
intervals  from  Appendix  I,  to  determine  if  any  detection  ranges  were  within  acceptable 
tolerance.  The  comparison  concluded  that  none  of  the  detection  ranges  were  within  the 
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acceptable  tolerance,  with  the  one  exception  noted  above.  (Figure  7  illustrates  the 
method  for  applying  the  A  value  to  the  confidence  interval  for  scenario  #3,  HL,  blue 
detection  range).  Additionally,  a  consistent  trend  in  the  detection  data  surfaced;  the  JLink 
detection  ranges  were  always  larger  than  the  Janus  detection  ranges,  for  both  the  blue  and 
red  forces. 


Figure  7.  Confidence  Intervals  Versus  Acceptance  Range 


The  next  step  was  to  identify  the  cause  of  the  disparate  results  and  the  trend  in  the 
data.  The  first  issue  investigated  was  the  passing  of  the  entity’s  z-coordinate.  If  the 
entity’s  z-coordinate  is  not  passed  properly  to  JLink,  or  interpreted  properly  by  JLink,  the 
entity  could  be  simulated  above  ground  level,  resulting  in  better  line  of  sight  and 
extended  detection  ranges.  This  hypothesis  turned  out  to  be  incorrect,  however,  because 
JLink  can  determine  which  entities  are  ground  entities.  When  the  information  about  a 
ground  entity  is  passed  to  JLink,  JLink  assigns  the  entity  the  x  and  y  coordinates  passed 
from  Janus  and  places  the  entity  on  the  ground.  The  ground  entity  then  uses  the  ground  z- 
coordinate  to  perform  line  of  sight  calculations. 
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The  second  proposed  solution  focused  on  the  fact  that  something  in  the  battlefield 
environment  was  affecting  the  detection  ranges.  The  issue  of  artillery  delivered  smoke 
and  munitions  was  suggested  as  a  possible  cause.  All  scenarios  used  large  volumes  of 
artillery  smoke  and  artillery  High  Explosive  (HE)  rounds.  As  discussed  in  the  analysis  of 
chemical  kills,  Janus  and  JLink  simulated  artillery  delivery  methods  differently  in  this 
experiment.  If  the  dimension  of  the  chemical  cloud  could  affect  the  number  of  chemical 
kills,  then  the  dimensions  of  any  cloud  generated  from  an  artillery  munition  may  also 
affect  the  range  at  which  an  entity  can  detect  through  the  cloud. 

To  test  the  proposed  solution,  the  SWA  armored  scenario  was  re-run,  excluding 
all  artillery  from  both  sides.  Thus,  in  this  case  there  were  no  clouds  resulting  from 
artillery  smoke  or  HE  rounds  on  the  battlefield.  The  hypothesis  was  that  the  artillery 
simulated  as  a  volley  (Janus)  led  to  decreased  visibility,  and  shorter  detection  ranges,  than 
artillery  simulated  as  individual  rounds  (JLink).  The  scenario  was  then  run  10  times  in 
both  the  Janus  and  JLINK  modes,  using  the  same  random  number  seeds.  The  results 
yielded  an  average  detection  range  for  red  Janus  entities  of  2.532  kilometers,  while  the 
average  detection  range  for  the  red  JLINK  entities  was  2.463  kilometers.  The  average 
detection  range  for  blue  Janus  entities  was  3.885  versus  3.767  for  the  blue  JLINK  entities. 
Table  34  depicts  the  actual  ranges  for  each  run.  The  assumption  of  normality  was 
satisfied,  but  the  F-Test  concluded  that  the  variances  for  the  populations  were  not  the 
same  for  the  blue  Janus  and  blue  JLink  detection  ranges.  The  variances  were  the  same, 
however,  for  the  red  Janus  and  red  JLink  detection  ranges.  Finally,  a  two-sample  /-test 
showed  that  in  both  cases,  the  means  for  the  populations  were  not  significantly  different. 
This  analysis  provides  a  basis  for  concluding  that  the  different  artillery  delivery  methods 
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contributed  to  the  disparate  detection  results.  Further,  this  thesis  concludes  that  the 


volley  method  used  by  Janus  to  deliver  artillery  munitions  generates  shorter  detection 


ranges  than  the  individual  round  method  due  to  the  dimensions  and  dissipation  rate  of  the 


cloud. 


Currently,  the  software  has  not  been  rewritten  in  JLink  to  fire  artillery  using  the 


same  method  as  Janus.  After  the  code  is  rewritten,  several  additional  experiments  will  be 


done  to  ensure  that  the  artillery  is  being  accurately  played  using  the  volley  method,  and 


that  this  method  generates  similar  detection  ranges  between  Janus  and  JLink. 


Det  Ran) 

*es  Blue 

Det  Ranges  Red 

RUN# 

Janus 

JLink 

Janus 

JLink 

1 

3.863 

3.785 

2.294 

2.711 

2 

3.940 

3.753 

2.513 

2.386 

3 

3.846 

3.846 

2.541 

2.541 

4 

3.778 

3.877 

2.408 

2.719 

5 

3.918 

3.918 

2.890 

2.889 

6 

4.017 

3.616 

2.912 

2.166 

7 

3.863 

3.567 

2.294 

2.319 

8 

3.912 

4.021 

2.715 

2.512 

9 

3.856 

3.660 

2.377 

2.275 

10 

3.853 

3.631 

2.377 

2.119 

XBAR 

3.885 

3.767 

•2.532 

2.464 

VAR 

0.004 

0.022 

0.054 

0.065 

STDEV 

0.065 

0.149 

0.232 

0.255 

t-Test  Stat 

0.040 

.515 

Crit.  Value 

2.101 

2.101 

Table  34.  Detection  Ranges  Without  Artillery 


D.  KILL  RANGES  (BLUE/RED) 

The  next  MOP  analyzed  was  kill  range,  analyzed  separately  by  blue  force  and  red 
force,  using  the  data  from  Appendix  B.  (The  headings  “Kill  Ranges  Blue/Red”  in 


Appendix  B  refer  to  the  those  units  being  killed).  The  results  of  the  two-sample  /-test  and 
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the  Wilcoxon  test  rejected  the  null  hypothesis  that  the  average  kill  ranges  (blue  and  red) 
are  same  between  modes  in  six  instances  : 

1 .  Blue  Kill  Ranges  in  Scenario  #1 ,  SWA 

2.  Blue  Kill  Ranges  in  Scenario  #1,  HL 

3.  Red  Kill  Ranges  in  Scenario  #1 ,  HL 

4.  Red  Kill  Ranges  in  Scenario  #2,  HL 

5.  Blue  Kill  Ranges  in  Scenario  #3,  SWA 

6.  Blue  Kill  Ranges  in  Scenario  #3,  HL 

Next,  the  acceptable  tolerance  range  of  A  =  0.2  kilometers  for  scenarios  1  and  2, 
and  A  =  0.05  kilometers  for  scenario  3,  was  compared  to  the  kill  range  CIs  in  Appendix  I 
for  those  scenarios  which  rejected  the  null  hypothesis.  The  results  of  the  comparison 
were  that  none  of  the  CIs  fell  within  the  acceptable  tolerance.  This  finding  can  be 
interpreted  by  stating  that  the  mean  kill  ranges  in  Janus  and  JLink  in  these  scenarios  are 
not  the  same,  and  indeed  so  dissimilar  that  the  difference  in  the  means  falls  outside  the 
acceptable  tolerance. 

As  with  the  detection  ranges  data,  a  consistent  trend  occurred  in  the  kill  range 
data;  the  kill  ranges  for  JLink  were  consistently  greater  than  the  kill  ranges  for  Janus. 

This  result  is  not  surprising  in  light  of  the  similar  trend  in  detection  ranges.  There  is  a 
strong  likelihood  that  if  an  entity  can  detect  at  further  range,  then  it  can  acquire  and  target 
at  greater  distances,  and  subsequently  kill  at  longer  ranges.  This  reasoning  is  based  on 
the  method  by  which  a  Janus  entity  performs  the  engagement  sequence.  After  conducting 
a  line  of  sight  calculation,  Janus  determines  the  target’s  location  and  identity.  Once 
identified,  Janus  determines  the  target’s  priority  and  places  it  on  the  target  list.  The  target 
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list  for  each  Janus  entity  determines  which  targets  will  be  engaged  and  in  what  order. 
Janus  then  acquires  the  target  and  engages.  The  engagements  may,  or  may  not,  lead  to  a 
kill.  This  entire  sequence,  however,  stems  from  the  initial  detection.  Therefore,  if  the 
detection  ranges  are  statistically  similar,  then  the  engagement  sequences  in  both  Janus 
and  JLink  may  be  in  agreement. 

The  link  between  detection  ranges  and  kill  ranges  focused  the  investigation  into 
resolving  the  disparate  kill  range  issue.  As  mentioned  in  the  previous  section,  further 
analysis  of  the  detection  ranges,  without  firing  artillery,  has  begun  using  the  armored 
scenario  in  SWA.  This  ongoing  investigation  will  also  analyze  the  kill  ranges  to  verify 
whether  the  kill  ranges  were  indeed  affected  by  the  detection  ranges.  The  assumption  is 
that  if  the  detection  range  issue  is  resolved,  the  kill  ranges  between  Janus  and  JLink  will 
become  more  similar.  Presently,  this  hypothesis  is  being  tested. 

The  last  result  analyzed  for  kill  ranges  is  the  evidence  of  interaction  between  the 
environment  and  mode  for  mean  red  kill  ranges.  The  interaction  plot  in  Appendix  E 
shows  that  the  red  force  has  the  advantage  in  JLink  of  greater  kill  ranges,  but  the 
advantage  is  even  more  pronounced  when  fighting  in  HL  terrain  as  compared  to  SWA 
terrain.  The  assumption  that  greater  detection  ranges  in  JLink  lead  to  greater  kill  ranges 
in  JLink  accounts  for  part  of  the  overall  advantage.  However,  all  other  factors  being 
equal,  one  would  expect  the  JLink  kill  range  advantage  to  be  consistent  between 
scenarios  since  the  JLink  detection  range  advantage  is  consistent  between  scenarios. 
Because  there  is  a  larger  red  kill  range  difference  in  the  HL  as  compared  to  SWA,  further 
investigation  is  needed  to  identify  the  factor  causing  the  difference. 


72 


One  likely  factor  causing  the  difference  in  red  kill  ranges  between  environments 
and  mode  is  the  different  terrain  features  of  HL  and  SWA.  Terrain  features  affect  line  of 
sight,  which  then  affect  detections,  and  subsequently  the  entire  engagement  process 
leading  to  rounds  fired.  Further  testing  should  focus  on  this  factor  by  simulating  battalion 
size  forces  attacking  under  these  two  different  environments.  The  issue  of  detection 
ranges  should  be  resolved  first,  however,  in  order  to  isolate  the  factors  causing 
differences  in  kill  ranges. 

Overall,  JLink’s  current  configuration  returns  kill  ranges  which  are  statistically 
similar  to  those  in  Janus  in  six  of  12  cases.  These  results  indicate  that  JLink  provides  a 
setting  which  generally  replicates  the  kill  ranges  of  different  types  of  units  fighting  in 
different  environments.  In  particular,  JLink  appears  quite  accurate  in  producing 
comparable  kill  ranges  for  armored  coalition  scenarios  fighting  on  the  SWA  terrain,  and 
attacking  light  infantry  forces  fighting  on  both  HL  and  SWA  terrain.  These  results  also 
indicate  that  the  JLink  battlefield  produces  fairly  accurate  kill  ranges  for  units  conducting 
different  types  of  missions,  in  this  case  an  attacking  red  force  against  a  defending  blue 
force.  Once  further  testing  is  complete,  and  the  issue  of  detection  range  is  resolved,  one 
would  expect  JLink  to  produce  kill  ranges  even  closer  to  those  of  Janus. 

E.  ROUNDS  FIRED  (BLUE/RED) 

The  next  MOP  analyzed  was  rounds  fired,  analyzed  separately  by  blue  force  and 
red  force,  using  the  data  from  Appendix  B.  The  ANOVA  returned  a  p-value  of  .96  for 
red  rounds  fired,  which  indicates  that  JLink  is  providing  a  battlefield  setting  which 
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replicates  the  average  number  of  rounds  fired  comparable  to  Janus  for  a  battalion-size 
armored  or  infantry  force  in  the  offense,  under  different  environmental  conditions. 

The  results  of  the  two-sample  /-test  and  the  Wilcoxon  test  for  blue  rounds  fired 
were  mutually  supporting,  although  the  /-test  was  not  performed  on  several  data  sets 
because  they  did  not  satisfy  r-test  assumptions,  primarily  due  to  wide  differences  in 
variance  between  populations.  The  tests  rejected  the  null  hypothesis  that  the  average  blue 
rounds  fired  are  the  same  between  modes  in  four  instances  : 

1.  Blue  Rounds  Fired  in  Scenario  #1,  SWA 

2.  Blue  Rounds  Fired  in  Scenario  #1,  HL 

3.  Blue  Rounds  Fired  in  Scenario  #2,  SWA 

4.  Blue  Rounds  Fired  in  Scenario  #3,  SWA 

The  ANOVA  supports  these  findings  as  indicated  by  the  F-values  for  Mode  in  Appendix 
E.  The  F-value  of  1 1.929  for  the  blue  rounds  fired  is  interpreted  as,  the  average  blue 
rounds  fired  among  all  Janus  scenarios  are  statistically  different  than  the  average  blue 
rounds  fired  among  all  JLink  scenarios. 

The  interaction  between  environment  and  mode  for  blue  rounds  fired  indicates 
that  the  difference  in  blue  rounds  fired  between  Janus  and  JLink  is  even  greater  when 
placed  in  the  SWA  environment.  As  mentioned  previously,  investigation  into  the  cause 
of  this  interaction  must  focus  on  the  competing  terrain  of  HL  and  SWA.  Terrain  affects 
line  of  sight  and  ultimately  the  engagement  sequence  and  rounds  fired.  Once  the 
detection  range  issue  is  resolved,  further  testing  should  focus  on  simulating  units  in  the 
defense  under  different  environments. 
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Next,  the  four  scenarios  which  rejected  the  null  hypothesis  were  analyzed  to 
determine  if  the  differences  in  the  number  of  rounds  fired  between  Janus  and  JLink 
satisfied  the  accepted  tolerance.  Recall  that  the  accepted  tolerance  for  rounds  fired  was 
A  =  10%  of  the  total  Janus  rounds  fired.  Applying  the  method  illustrated  in  Figure  7  of 
comparing  the  MOP’s  confidence  interval  to  the  determined  A  -  value  resulted  in  none  of 
the  four  being  within  acceptable  tolerance. 

Based  on  the  results  of  the  statistical  tests  for  rounds  fired,  JLink  provides  an 
accurate  replication  of  the  battlefield  that  generates  an  average  number  of  rounds  fired 
comparable  to  Janus  for  a  battalion-size  unit  attacking  under  different  environmental 
conditions.  JLink  also  provides  a  fare  replication  of  the  battlefield  that  generates  similar 
average  rounds  fired  for  a  company-size  unit  defending  in  HL  terrain. 

Further,  in  reference  to  the  discussion  of  the  engagement  process  in  paragraph  D 
above,  the  number  of  rounds  fired  could  become  more  similar  between  Janus  and  Jlink 
once  the  detection  range  and  kill  range  issues  are  resolved.  Since  JLink  consistently  had 
greater  detection  ranges  and  kill  ranges  than  Janus  in  this  experiment,  one  would 
conclude  that  the  number  of  rounds  fired  were  affected  by  the  disparity  given  that  all 
three  MOPs  are  linked  in  the  engagement  process.  The  ongoing  experiment  of  the 
armored  scenario  in  SWA,  stripped  of  all  artillery  munitions,  will  be  a  valuable  study  to 
analyze  the  effect  that  accurate  detection  ranges  have  on  rounds  fired,  especially  since 
both  the  blue  and  red  forces  in  the  armored  scenario  in  SWA  appeared  to  have  different 
averages  of  numbers  of  rounds  fired. 
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F.  LOSS  EXCHANGE  RATIO  (LER) 

The  LER  is  the  ratio  of  red  casualties  to  blue  casualties,  and  offers  a  quantitative 
method  to  assess  the  overall  outcome  of  a  battle.  Given  that  the  factors  affecting  the  ratio 
are  accurately  portrayed,  LERs  determined  to  be  statistically  similar  provide  strong 
supporting  evidence  that  JLink  is  accurately  portraying  the  entire  scenario.  In  this  case, 
both  the  r-test  and  Wilcoxon  test  failed  to  reject  the  null  hypothesis  in  four  of  the  six 
scenarios  when  testing  the  LERs.  This  indicates  that  the  Janus  and  JLink  LERs  for  these 
four  scenarios  are  similar.  In  scenario  #1,  SWA  and  scenario  #3,  SWA,  the  hypothesis  of 
equal  LERs  was  rejected.  Although,  on  four  of  six  occasions,  the  null  hypothesis  was  not 
rejected,  an  analyst  must  be  cautious  to  accept  the  conclusion  that  JLink  is  accurately 
portraying  the  overall  battle  under  these  four  conditions  based  on  these  results. 

The  question  posed  then  is,  why  are  two  of  the  six  not  statistically  similar?  The 
LER  is  a  function  of  several  other  factors,  primarily  kills  and  detections.  Previous 
analysis  has  shown  that  kills  due  to  chemical  artillery  were  not  accurately  portrayed  by 
JLink  during  the  experiment.  Chemical  kills  is  a  factor  of  the  LER,  and  inaccuracies  with 
simulating  those  kills  will  adversely  affect  the  LER  for  that  scenario.  Additionally, 
discussion  of  the  detection  ranges  concluded  that  JLink  did  not  satisfactorily  replicate  the 
detection  ranges  in  this  experiment,  which  is  the  another  major  contributing  factor  of  the 
LER.  The  chemical  kills  and  detection  range  issues  alone  could  lead  to  dissimilar  LERs 
between  Janus  and  JLink. 

Given  that  at  least  two  inputs  to  the  LER  equation  may  not  have  been  accurate,  it 
is  reassuring  that  four  of  the  six  combinations  were  within  statistical  tolerance.  There  is  a 
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strong  likelihood  that  once  the  artillery  delivery  methods  in  JLink  are  identical  to  Janus, 
the  ongoing  investigations  into  chemical  kills  and  detection  ranges  may  be  resolved. 
After  these  issues  are  resolved,  this  MOP  should  be  analyzed  again.  One  would  expect 
the  LERs  to  become  more  similar  for  all  scenarios  once  these  major  factors  in  the  LER 
equation  are  corrected. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


This  chapter  draws  conclusions  from  Chapter  V,  assessing  the  current  state  of 
JLink  in  terms  of  varied  scenarios,  coalition  warfare,  and  specific  combat  functions.  The 
chapter  also  provides  recommendations  for  further  study  of  JLink  issues  presented  in  this 
thesis. 

A.  CONCLUSIONS 

1.  Varied  Scenarios 

Chapter  II  describes  varied  scenarios  as  different  types  of  combat  units  in  the 
offense  and  defense,  negotiating  obstacles,  under  different  environmental  conditions. 

Such  scenarios  had  yet  to  be  tested  in  JLink  to  the  extent  tested  in  this  thesis.  To  assess 
whether  JLink  replicates  these  scenarios  in  a  way  similar  to  Janus,  the  MOPs  detection 
range,  kill  range,  rounds  fired,  and  LER  were  analyzed.  The  first  three  MOPs  allowed  for 
analysis  of  the  basic  procedures  of  an  engagement,  which  is  critical  to  arriving  at  valid 
conclusions  about  scenario  fidelity.  The  LER  offered  a  quantitative  method  to  assess  the 
overall  scenario. 

Although  JLink  did  not  accurately  replicate  detection  ranges  in  this  experiment, 
the  related  MOPs  of  kill  range  and  rounds  fired  fared  well  in  the  overall  analysis.  Also, 
statistical  tests  concluded  that  four  of  the  six  JLink  LERs  were  statistically  similar  to  the 
corresponding  Janus  LERs.  These  results  support  the  conclusion  that  JLink  is  robust 
enough  to  sufficiently  replicate  varied  scenarios  in  a  manner  similar  to  Janus,  despite 
inaccurate  inputs  in  detection  ranges. 
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2. 


Coalition  Warfare 


The  same  MOPs  used  for  drawing  conclusions  about  varied  scenarios 
were  used  to  assess  JLink’s  ability  to  portray  coalition  warfare.  Notwithstanding  the 
detection  range  inaccuracies,  all  MOPs  analyzed  for  scenario  #2,  armored  coalition  in 
both  HL  and  SWA  environments,  failed  to  reject  the  null  hypothesis,  except  for  red  kill 
ranges  in  HL  and  blue  rounds  fired  in  SWA.  Given  these  results,  the  thesis  concludes 
that  JLink  produces  similar  results  to  stand-alone  Janus  when  simulating  coalition 
warfare. 

3.  Specific  Combat  Functions 

a.  FASCAM 

The  MOP  used  to  assess  JLink  FASCAM  fidelity  was  the  number  of  kills 
resulting  from  FASCAM  minefields.  Although  the  statistical  results  for  FASCAM 
rejected  the  null  hypothesis  only  once,  the  trend  towards  consistently  higher  Janus 
FASCAM  kills  indicates  that  Janus  FASCAM  may  be  more  lethal  than  JLink  FASCAM. 
Despite  the  trend,  however,  JLink  appears  to  provide  a  fair  representation  of  FASCAM. 
A  stronger  conclusion  must  be  borne  out  of  additional  FASCAM  testing  which  isolates 
the  minefield. 

b.  Chemical  Artillery 

The  MOP  used  to  assess  JLink  chemical  artillery  was  the  number  of  kills 
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generated  by  chemical  artillery  rounds.  In  all  cases,  the  statistical  tests  rejected  the  null 
hypothesis  that  the  average  number  of  Janus  chemical  kills  were  the  same  as  the  average 
number  of  JLink  chemical  kills.  Given  these  results,  the  thesis  concludes  that  JLink’s 
configuration  during  the  experiment  did  not  accurately  replicate  the  effects  of  Janus 
chemical  artillery. 

The  investigation  into  the  cause  of  the  disparate  chemical  data  resulted  in  a 
solution  which  changed  the  delivery  of  JLink  chemical  artillery  from  an  individual  round 
method  to  a  volley  method.  Changing  to  the  volley  method  and  rerunning  the  armored 
scenario  in  SWA  produced  very  similar  chemical  kills  between  Janus  and  JLink.  The 
conclusion  is  that  JLink  replicates  the  effects  of  Janus  chemical  artillery  for  an  armored 
scenario  in  SWA  terrain  when  fired  in  volley.  Further,  the  volley  method  must  be  used 
whenever  a  user  simulates  a  JLink  to  JLink  battle. 

B.  RECOMMENDATIONS 

1.  Artillery  Code  Upgrade 

After  investigating  the  cause  of  the  detection  range  disparities,  a  brief  study  was 
conducted  without  artillery  to  support  the  hypothesis  that  different  artillery  delivery 
methods  contributed  to  the  different  detection  results.  The  study,  which  involved  the 
armored  scenario  in  SWA,  proved  conclusive.  Without  artillery,  the  detection  ranges 
were  statistically  similar.  The  conclusion  is  that  different  artillery  delivery  methods 
produce  different  detection  ranges. 

The  recommendation  is  that  JLink  follow  the  volley  method  used  in  Janus 
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stand-alone,  and  that  code  be  rewritten  in  JLink  to  fire  artillery  using  the  volley  method 
when  two  JLink  players  are  involved  in  a  simulation. 

2.  Further  Testing  of  Detection  Ranges,  Kill  Ranges,  Rnds  Fired,  LERs 

Once  the  JLink  code  is  rewritten  using  the  volley  method  for  artillery,  further  tests 
are  necessary  using  various  scenarios  and  environments  to  confirm  that  JLink  scenarios 
are  generating  similar  detection  ranges  as  compared  to  Janus  scenarios. 

Also,  the  kill  ranges  for  these  scenarios  should  be  analyzed  to  ensure  JLink 
scenarios  are  producing  similar  kill  ranges  and  LERs  as  compared  to  Janus.  The 
assumption  is  that  detection  ranges  significantly  affect  kill  ranges,  based  on  the 
discussion  of  the  engagement  sequence.  Once  the  detection  issue  is  resolved,  one  would 
expect  the  kill  ranges  to  be  more  aligned  between  Janus  and  JLink.  Then,  further  testing 
should  focus  on  the  kill  range  results  of  a  battalion-size  unit  attacking  in  the  HL  and 
SWA  terrain  in  order  to  identify  the  source  of  interaction  between  environment  and 
mode. 

Rounds  fired  should  also  be  analyzed  in  the  re-run  scenarios.  Again,  given  the 
engagement  sequence,  one  would  expect  the  average  rounds  fired  to  be  more  similar 
between  Janus  and  JLink  once  detection  ranges  and  kill  ranges  are  corrected.  Additional 
testing  should  focus  on  units  in  the  defense  under  both  environments  to  isolate  the  cause 
of  environment  and  mode  interaction  for  blue  rounds  fired. 

Since  detection  ranges,  kill  ranges,  and  rounds  fired  are  inputs  into  the  LER 
equation,  one  would  expect  the  JLink  LERs  to  be  more  in  line  with  the  Janus  LERs  once 
the  detection  ranges  and  kill  ranges  generate  similar  results  between  the  two  modes. 
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Testing  and  resolving  these  issues  will  lead  to  a  more  robust  JLink  system,  capable  of 
accurately  portraying  more  diverse  scenarios. 

3.  Further  Testing  of  FASCAM  and  Chemical  Artillery 

The  trend  in  the  data  consistently  showing  greater  Janus  FASCAM  kills  requires 
that  FASCAM  be  investigated  further.  Also,  chemical  artillery  must  be  tested  using 
scenarios  other  than  the  previously  tested  armored  scenario  in  SWA  to  confirm  that  the 
JLink  code  upgrades  for  chemical  rounds  sufficiently  replicates  chemical  kills  between 
the  two  modes.  Improvements  in  both  FASCAM  and  chemical  artillery  will  diversify 
JLink  and  increase  its  value  as  a  training  and  analytical  tool. 

C.  FINAL  COMMENTS 

This  thesis  provides  the  latest  assessment  of  JLink  fidelity.  The  conclusions  and 
recommendations  of  this  thesis  will  improve  distributed  interoperability.  However,  in 
order  to  achieve  the  final  goal  of  100%  interoperability  between  JLink  systems,  continued 
testing  is  necessary  on  the  topics  mentioned  in  this  thesis  and  future  topics  generated  by 
the  evolving  Jlink  system. 
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APPENDIX  A.  RAW  DATA 
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36 

2.59 

1.32 

mm 

1.770 

mi 

1.511 

2284 

2978 

8 

1 

1 

69 

30 

2.42 

2.35 

BH 

1.725 

1.416 

2816 

3246 

9 

3 

0 

69 

39 

2.93 

2.9 

MBm 

1.664 

ms 

1.345 

2727 

2565 

10 

1 

0 

69 

51 

2.69 

1.53 

■m 

1.733 

MEM 

1.567 

2705 

2782 

XBAR 

1 

0.6 

69.6 

36.6 

3.112 

2.36 

1.610 

1.701 

1.313 

1.422 

2713.1 

2935.5 

VAR 

1.11 

0.27 

1.6 

77.6 

1.004 

0.436 

0.001 

0.001 

0.003 

0.007 

88588.1 

55293.61 

STDEV 

1.05 

0.52 

1.26 

8.81 

1.002 

0.660 

0.029 

0.037 

0.054 

0.082 

297,64 

235.15 
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APPENDIX  B.  RAW  DATA  SEPARATED  BY  SIDE 


SCENARIO  #1,  SWA  RESULTS 


Det  Ranges  Blue 

Det  Ranges  Red 

Kilt  Ranges  Blue 

Kill  Ranges  Red 

Rnds  Fired  Blue 

Rnds  Fired  Red  | 

Run# 

Janus 

JUNK 

Janus 

JLINK 

Janus 

JLINK 

Janus 

JLINK 

Janus 

JUNK 

Janus 

JUNK 

1 

3.847 

4.231 

2.381 

3.398 

1.986 

2.356 

2.704 

2.987 

60 

27 

176 

221 

3.707 

4.139 

2.344 

3.393 

1.903 

2.413 

2.805 

3.028 

54 

23 

65 

212 

3.616 

4.220 

2.412 

3.510 

1.952 

2.343 

2.901 

3.064 

172 

29 

75 

172 

3.656 

4.274 

2.480 

3.425 

2.020 

2.393 

3.153 

3.404 

87 

29 

96 

134 

3.570 

4.164 

2.661 

3.452 

1.862 

2.559 

2.725 

2.826 

102 

23 

153 

168 

3.825 

4.068 

2.308 

3.538 

1.905 

2.353 

3.083 

2.930 

63 

55 

87 

247 

3.570 

4.245 

2.593 

3.502 

1.929 

2.381 

2.661 

2.909 

108 

30 

117 

198 

8 

3.651 

4.187 

2.308 

3.108 

3.049 

2.458 

2.613 

3.630 

85 

30 

120 

193 

9 

3.764 

4.087 

2.414 

3.418 

2.039 

2.404 

2.860 

3.037 

64 

38 

123 

208 

10 

3.684 

4.094 

2.800 

3.148 

2.096 

2.276 

2.968 

3.185 

58 

44 

89 

200 

XBAR 

3.689 

4.171 

2.470 

3.389 

2.074 

2.394 

2.847 

3.100 

85.3 

32.8 

110.1 

195.3 

VAR 

0.010 

0.005 

0.027 

0.021 

0.123 

0.006 

0.033 

0.061 

1292.23 

101.73 

1215.43 

983.79 

STDEV 

0.098 

0.072 

0.165 

0.146 

0.350 

0.076 

0.181 

0.246 

35.95 

10.09 

34.86 

31.37 

SCENARIO  #1,  HL  RESULTS 


Det  Ranges  Blue 

Det  Ranges  Red 

Kill  Ranges  Blue 

Kill  Ranges  Red 

Rnds  Fired  Blue 

Rnds  Fired  Red  j 

Run# 

Janus 

JUNK 

Janus 

JUNK 

Janus 

JLINK 

Janus 

JLINK 

Janus 

JLINK 

Janus 

JUNK 

1 

3.389 

3.845 

2.022 

2.959 

2.000 

2.790 

2.629 

3.112 

171 

65 

375 

263 

3.419 

4.047 

2.076 

3.078 

1.821 

2.302 

2.835 

3.003 

87 

59 

180 

197 

3.419 

4.060 

2.068 

3.092 

2.249 

2.580 

2.590 

3.060 

150 

65 

304 

3.259 

3.970 

2.142 

2.886 

2.138 

3.019 

2.534 

2.889 

78 

59 

737 

492 

3.362 

3.901 

1.880 

3.086 

1.920 

2.689 

2.534 

149 

60 

421 

3.211 

3.913 

2.070 

2.530 

1.726 

2.293 

2.470 

147 

62 

271 

292 

3.258 

4.060 

2.103 

2.797 

1.851 

2.427 

2.740 

2.866 

86 

49 

619 

361 

8 

3.349 

3.765 

1.892 

2.686 

1.617 

2.601 

2.410 

3.000 

87 

62 

582 

391 

9 

3.338 

4.071 

1.849 

2.835 

2.248 

2.237 

2.664 

3.014 

90 

53 

749 

267 

10 

3.296 

3.873 

1.903 

2.871 

1.978 

2.623 

2.670 

3.045 

78 

47 

1081 

337 

XBAR 

3.951 

2.000 

2.997 

112.3 

58.1 

531.9 

323.8 

VAR 

0.005 

0.012 

0.012 

0.034 

0.045 

0.061 

0.016 

0.007 

1360.01 

40.32 

75729.21 

6535.07 

STDEV 

0.072 

0.107 

0.108 

0.183 

0.212 

0.246 

0.127 

0.084 

36.88 

6.35 

275.19 

80.84 

SCENARIO  #2,  SWA  RESULTS 


Det  Ranges  Blue 

Det  Ranges  Red 

Kill  Ranges  Blue 

Kill  Ranges  Red 

Rnds  Fired  Blue 

Rnds  Fired  Red  | 

Run# 

Janus 

JUNK 

Janus 

JUNK 

Janus 

JUNK 

Janus 

JUNK 

Janus 

JUNK 

Janus 

JLINK 

1 

3.841 

4.128 

2.316 

3.271 

1.871 

2.098 

3.277 

3.115 

59 

35 

82 

112 

2 

3.718 

4.118 

2.218 

3.365 

1.788 

2.205 

3.019 

3.026 

34 

74 

102 

171 

3 

3.685 

3.992 

2.292 

3.325 

2.687 

2.098 

2.796 

2.958 

42 

29 

173 

160 

4 

3.614 

4.075 

2.219 

3.166 

1.742 

2.197 

2.993 

2.902 

49 

32 

91 

199 

5 

3.584 

4.151 

2.115 

3.376 

2.672 

2.179 

2.840 

3.162 

72 

38 

79 

189 

6 

3.173 

4.090 

2.030 

2.998 

1.766 

2.246 

3.173 

2.838 

52 

28 

335 

124 

7 

3.784 

4.103 

2.328 

3.219 

1.864 

2.121 

2.929 

2.901 

74 

36 

108 

192 

8 

3.611 

4.090 

2.091 

3.234 

1.771 

2.110 

2.900 

2.843 

62 

30 

88 

154 

9 

3.558 

4.176 

2.208 

3.492 

1.790 

2.375 

3.179 

3.609 

70 

22 

234 

158 

10 

3.625 

3.812 

2.302 

2.977 

1.916 

3.626 

2.940 

2.684 

51 

29 

148 

198 

XBAR 

3.619 

4.073 

2.212 

3.242 

1.987 

2.325 

3.005 

3.004 

56.5 

35.3 

144 

165.7 

VAR 

0.033 

0.011 

0.011 

0.027 

0.136 

0.216 

0.025 

0.065 

176.5 

206.01 

6945.78 

918.46 

STDEV 

0.181 

0.104 

0.103 

0.163 

0.369 

0.465 

0.158 

0.255 

13.29 

14.35 

83.34 

30.31 
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SCENARIO  #2,  HL  RESULTS 


Det  Ranges  Blue 

Det  Ranges  Red 

Kill  Ranges  Blue 

Kill  Ranges  Red 

Rnds  Fired  Blue 

Rnds  Fired  Red  | 

Run  # 

Janus 

JLINK 

Janus 

JLINK 

Janus 

JLINK 

Janus 

JLINK 

Janus 

JLINK 

Janus 

JLINK 

1 

3.312 

3.900 

1.944 

2.913 

2.033 

1.738 

2.423 

3.164 

78 

91 

307 

262 

3.573 

4.036 

1.907 

3.021 

2.073 

2.226 

2.797 

3.028 

41 

60 

475 

302 

3.361 

3.991 

1.792 

2.577 

2.011 

1.741 

2.500 

3.145 

64 

59 

543 

393 

3.301 

3.982 

1.866 

2.648 

1.683 

2.066 

2.737 

2.932 

83 

67 

367 

245 

3.465 

3.904 

1.899 

2.709 

1.984 

2.599 

2.769 

2.963 

56 

81 

422 

439 

3.245 

3.890 

2.025 

2.496 

1.718 

2.275 

2.541 

2.908 

67 

123 

524 

588 

3.401 

4.074 

1.807 

2.474 

1.536 

2.544 

2.599 

3.112 

66 

56 

729 

288 

8 

3.338 

3.970 

1.839 

2.717 

1.612 

2.561 

2.492 

2.967 

116 

58 

697 

630 

9 

3.335 

3.838 

1.993 

2.227 

1.582 

1.887 

2.633 

2.970 

63 

72 

648 

450 

10 

3.286 

3.895 

1.806 

2.463 

2.083 

1.809 

2.765 

2.927 

82 

69 

522 

522 

XBAR 

3.362 

3.948 

1.888 

1.831 

2.145 

3.012 

71.6 

73.6 

523.4 

411.9 

VAR 

0.009 

0.006 

0.007 

0.054 

0.050 

0.120 

0.018 

0.009 

401.6 

424.04 

19137.16 

18999.88 

STDEV 

0.096 

0.074 

0.081 

0.232 

0.224 

0.346 

0.135 

0.095 

20.04 

20.59 

138.34 

137.84 

SCENARIO  #3,  SWA  RESULTS 


Det  Ranges  Blue 

Det  Ranges  Red 

Kill  Ranges  Blue 

Kill  Ranges  Red 

Rnds  Fired  Blue 

Rnds  Fired  Red  | 

Run  # 

Janus 

JLINK 

Janus 

JLINK 

Janus 

JLINK 

Janus 

JLINK 

JLINK 

Janus 

JLINK 

1 

2.050 

2.038 

1.811 

2.034 

1.359 

1.616 

wsm 

1.427 

875 

646 

1750 

2 

2.071 

2.070 

mmm 

1.990 

1.481 

1.611 

mBm 

1.433 

866 

628 

mBm 

1625 

3 

2.057 

2.116 

mBm 

1.992 

1.596 

1.594 

■ EH 

1.365 

972 

557 

mm 

1645 

4 

2.014 

2.068 

Wmm 

2.052 

1.372 

1.683 

119 

1.370 

1295 

654 

1655 

5 

2.046 

2.034 

BS9 

1.989 

1.450 

1.635 

na 

1.271 

1137 

638 

mSm 

1525 

6 

2.087 

2.107 

mEm 

2.007 

1.525 

1.609 

1.244 

1.495 

833 

546 

WBSm 

1565 

7 

2.027 

2.079 

mBm 

1.973 

1.510 

1.610 

1.366 

1.315 

1456 

617 

— 

1675 

8 

2.058 

2.075 

mBm 

2.028 

1.540 

1.631 

1.345 

1.338 

749 

601 

III 

1555 

9 

2.126 

2.066 

111 

2.048 

1.462 

1.659 

1.416 

1.313 

658 

637 

— 

1570 

10 

2.138 

2.097 

1.817 

1.971 

1.476 

1.611 

1.286 

1.365 

855 

689 

■■ 

1625 

XBAR 

2.075 

1.791 

2.008 

1.313 

1.369 

969.6 

621.3 

1626 

1619 

VAR 

0.002 

0.001 

0.001 

0.001 

0.005 

0.001 

0.005 

0.004 

63081.38 

1896.46 

11621.11 

4510 

STDEV 

0.040 

0.027 

0.030 

0.030 

0.072 

0.027 

0.067 

0.067 

251.16 

43.55 

107.80 

67.16 

SCENARIO  #3,  HL  RESULTS 


Det  Ranges  Blue 

Det  Ranges  Red 

Kill  Ranges  Blue 

Kill  Ranges  Red 

Rnds  Fired  Blue 

Rnds  Fired  Red  | 

Run  # 

Janus 

JLINK 

Janus 

JLINK 

Janus 

JLINK 

Janus 

JLINK 

Janus 

JLINK 

Janus 

JLINK 

1 

1.613 

1.718 

1.547 

1.702 

1.327 

1.377 

1.322 

1.474 

1960 

1276 

855 

1620 

2 

1.641 

1.707 

1.572 

1.652 

1.312 

1.359 

1.287 

1.412 

1068 

1303 

1315 

1410 

3 

1.665 

1.703 

1 .586 

1.634 

1.312 

1.363 

1.300 

1.378 

1345 

1535 

1550 

1775 

4 

1.675 

1.676 

1.612 

1.626 

1.193 

1.420 

1.364 

1.275 

1833 

1636 

1475 

1285 

5 

1.614 

1.733 

1.536 

1.690 

1.319 

1.336 

1.202 

1.572 

1257 

1361 

1155 

1760 

6 

1.723 

1.677 

1.579 

1.651 

1.323 

1.376 

1.394 

1.366 

1151 

1268 

1635 

1555 

7 

1.632 

1.786 

1.538 

1.744 

1.399 

1.373 

1.376 

1.617 

1134 

1018 

1150 

1960 

8 

1.644 

1.747 

1.547 

1.688 

1.241 

1.362 

1.253 

1.438 

1131 

1486 

1685 

1760 

9 

1.623 

1.690 

1.543 

1.620 

1.299 

1.398 

1.403 

1.327 

1352 

1330 

1375 

1235 

10 

1.646 

1.753 

1.530 

1.709 

1 .286 

1.386 

1.265 

1.687 

1245 

1202 

1460 

1580 

XBAR 

1.648 

1.719 

1.559 

1.672 

1.301 

1.375 

1.317 

1.455 

1347.6 

1341.5 

1365.5 

1594 

VAR 

0.001 

0.001 

0.001 

0.002 

0.003 

0.001 

0.004 

0.018 

93197.38 

31205.83 

64852.5 

53604.44 

STDEV 

0.033 

0.036 

0.027 

0.041 

0.055 

0.023 

0.067 

0.133 

305.28 

176.65 

254.66 

231.53 
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APPENDIX  C.  RAW  DATA  FOR  SAME  SEED  JLINK  RUNS 


SCENARIO  #1 ,  SWA  JLINK  RESULTS 


Seed# 

Run# 

FASCAM  Kills 

Chem  Kills 

LER 

Detection  Ranges 

Kill  Ranges 

Rounds  Fired 

55 

1 

4 

0 

1.08 

3.844 

2.636 

248 

55 

2 

1 

1 

1 

3.816 

2.707 

260 

55 

3 

3 

0 

1.14 

3.851 

2.774 

158 

55 

4 

1 

0 

2.09 

3.705 

2.804 

252 

55 

5 

2 

0 

1.13 

3.853 

2.851 

159 

55 

6 

1 

0 

1.3 

3.786 

2.635 

182 

55 

7 

0 

0 

1.17 

3.809 

2.711 

257 

55 

8 

2 

0 

0.88 

3.822 

2.645 

196 

55 

9 

1 

0 

0.92 

3.837 

2.604 

224 

55 

10 

4 

0 

1.17 

3.956 

2.700 

230 

XBAR 

1.9 

0.1 

1.188 

3.828 

216.6 

VAR 

1.88 

0.1 

0.116 

0.004 

0.007 

1595.82 

STDEV 

1.37 

0.32 

0.341 

0.063 

0.081 

39.95 

SCENARIO  #1,  HL  JUNK  RESULTS 


Seed# 

Run# 

FASCAM  Kills 

Chem  Kills 

LER 

Detection  Ranges 

Kill  Ranges 

Rounds  Fired 

55 

1 

3 

3 

1.69 

3.544 

2.983 

328 

55 

2 

1 

1 

1.21 

3.531 

2.757 

377 

55 

3 

2 

3 

1.32 

3.583 

2.776 

248 

55 

4 

2 

2 

1.56 

3.624 

2.983 

219 

55 

5 

0 

2 

1.43 

3.787 

3.120 

306 

55 

6 

2 

2 

1.1 

3.633 

2.763 

466 

55 

7 

1 

4 

1,46 

3.438 

2.805 

307 

55 

8 

2 

4 

1.91 

3.583 

2.811 

302 

55 

9 

1 

1 

1.15 

3.689 

2.617 

212 

55 

10 

2 

3 

1.92 

3.691 

2.811 

277 

XBAR 

1.6 

2.5 

1.475 

3.610 

2.843 

304.2 

VAR 

0.71 

1.17 

0.087 

0.010 

0.021 

5746.62 

STDEV 

0.84 

1.08 

0.295 

0.098 

0.145 

75.81 

SCENARIO  #2,  SWA  JLINK  RESULTS 


Seed# 

Run  # 

FASCAM  Kills 

Chem  Kills 

LER 

Detection  Ranges 

Kill  Ranges 

Rounds  Fired 

55 

1 

3 

0 

1.4 

3.364 

2.635 

147 

55 

2 

0 

0 

1.17 

3.369 

2.859 

106 

55 

3 

0 

0 

1.19 

3.360 

2.441 

176 

55 

4 

2 

1 

1.2 

3.514 

2.587 

158 

55 

5 

2 

0 

1.35 

3.428 

2.550 

204 

55 

6 

2 

0 

1.26 

3.451 

3.480 

121 

55 

7 

0 

0 

1.32 

3.223 

2.500 

258 

55 

8 

1 

1 

1.17 

3.283 

2.831 

243 

55 

9 

1 

0 

0.96 

3.324 

2.539 

256 

55 

10 

0 

1 

1.31 

3.339 

2.467 

319 

XBAR 

1.1 

0.3 

1.233 

3.365 

2.689 

198.8 

VAR 

1.21 

0.23 

0.016 

0.007 

0.097 

4748.62 

STDEV 

1.10 

0.48 

0.125 

0.084 

0.312 

68.91 
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SCENARIO  #2,  HL  JUNK  RESULTS 


Seed  # 

Run  # 

FASCAM  Kills 

Chem  Kills 

LER 

Detection  Ranges 

Kill  Ranges 

Rounds  Fired 

55 

1 

0 

2 

2.4 

2.398 

2.745 

353 

55 

2 

2 

1 

1.36 

2.254 

2.501 

527 

55 

3 

1 

1 

1.5 

2.189 

2.619 

362 

55 

4 

2 

2 

1.63 

2.469 

2.692 

319 

55 

5 

2 

0 

1.31 

2.264 

2.687 

426 

55 

6 

2 

3 

1.77 

2.435 

2.792 

450 

55 

7 

2 

4 

2.63 

2.486 

2.787 

461 

55 

8 

0 

1 

1.54 

2.518 

2.712 

336 

55 

9 

2 

2 

1.59 

2.284 

2.708 

531 

55 

10 

3 

2 

1.79 

2.415 

2.498 

549 

XBAR 

1.6 

1.8 

1.752 

2.371 

2.674 

431.4 

VAR 

0.93 

1.29 

0.188 

0.013 

0.011 

7426.49 

STDEV 

0.97 

1.14 

0.433 

0.114 

0.105 

86.18 

SCENARIO  #3,  SWA  JLINK  RESULTS 


Seed  # 

Run# 

FASCAM  Kills 

Chem  Kills 

LER 

Detection  Ranges 

Kill  Ranges 

Rounds  Fired 

55 

1 

0 

0 

0.94 

2.036 

1.526 

2396 

55 

2 

0 

0 

1.18 

2.030 

1.423 

2268 

55 

3 

0 

0 

0.87 

2.059 

1.508 

2126 

55 

4 

0 

0 

0.78 

2.053 

1.511 

2228 

55 

5 

0 

3 

1.03 

2.057 

1.413 

2212 

55 

6 

1 

0 

1.09 

2.060 

1.441 

2285 

55 

7 

0 

0 

0.95 

2.055 

1.482 

2340 

55 

8 

0 

0 

0.95 

2.088 

1.564 

2321 

55 

9 

0 

0 

0.95 

2.022 

1.520 

2215 

55 

10 

0 

0 

1 

2.046 

1.488 

2215 

XBAR 

0.1 

0.3 

0.974 

2.051 

2260.6 

VAR 

0.1 

0.9 

0.012 

0.000 

0.002 

6070.71 

STDEV 

0.32 

0.95 

0.111 

0.019 

0.049 

77.91 

SCENARIO  #3,  HL  JLINK  RESULTS 


Seed  # 

Run# 

FASCAM  Kills 

Chem  Kills 

LER 

Detection  Ranges 

Kill  Ranges 

Rounds  Fired 

55 

1 

1 

33 

2.22 

1.713 

1.443 

2896 

55 

2 

1 

36 

2.29 

1.726 

1.481 

3137 

55 

3 

2 

42 

2.8 

1.735 

1.414 

3455 

55 

4 

0 

48 

2.81 

1.724 

1.420 

3066 

55 

5 

0 

33 

1.52 

1.738 

1.548 

2592 

55 

6 

0 

24 

1.58 

1.799 

1.493 

2835 

55 

7 

0 

36 

1.93 

1.696 

1.377 

2872 

55 

8 

2 

30 

2.19 

1.686 

1.322 

3085 

55 

9 

0 

33 

1.95 

1.692 

1.402 

2839 

55 

10 

0 

21 

1.87 

1.735 

1.406 

2798 

XBAR 

0.6 

33.6 

2.116 

1.724 

1.431 

2957.5 

VAR 

0.71 

61.6 

0.196 

0.001 

0.004 

56325.17 

STDEV 

0.84 

7.85 

0.442 

0.032 

0.064 

237.33 
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APPENDEX  D.  POWER  DATA 


Scenario  #1,  SWA 

D 

sp 

♦ 

Power 

FASCAM 

1 

0.070 

22.54 

1.00 

Chemical  Kills 

2 

0.601 

5.26 

1.00 

LER 

.3 

0.271 

1.75 

.72 

Detection  Range 

.3 

0.070 

6.76 

1.00 

Kill  Range 

.2 

0.136 

2.32 

.88 

Rounds  Fired 

40 

42.262 

1.50 

.54 

Scenario#!,  HL 

O 

Power 

FASCAM 

1 

1.362 

1.16 

.31 

Chemical  Kills 

2 

1.502 

2.11 

.81 

LER 

.3 

0.320 

1.48 

.54 

Detection  Range 

.3 

0.127 

3.73 

1.00 

Kill  Range 

.2 

0.124 

2.56 

.93 

Rounds  Fired 

130 

188.458 

1.09 

.29 

Scenario  #2,  SWA 

□ 

sp 

♦ 

Power 

FASCAM 

1 

1.464 

1.08 

.28 

Chemical  Kills 

2 

0.5 

6.33 

1.00 

LER 

.3 

0.219 

2.17 

.83 

Detection  Range 

.3 

0.091 

5.19 

1.00 

Kill  Range 

.2 

0.200 

1.58 

.56 

Rounds  Fired 

40 

63.909 

1 

0.99 

.25 
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APPENDIX  E.  ANALYSIS  OF  VARIANCE  (ANOVA)  RESULTS 


ANOVA  FOR  FASCAM  KILLS 

Analysis  of  Variance  Table 
Response:  MOE 

Terms  added  sequentially  (first  to  last) 

Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 

Seen  2  49.7167  24.85833  15.62689  0.0000011 

Env  1  0.3000  0.30000  0.18859  0.6649580 

Mode  1  32.0333  32.03333  20.13737  0.0000181 

Seen : Env  2  1.0500  0.52500  0.33003  0.7196211 

Seen : Mode  2  2.8167  1.40833  0.88533  0.4155499 

Env : Mode  1  1.6333  1.63333  1.02678  0.3131820 

Seen : Env : Mode  2  3.8167  1.90833  1.19965  0.3052820 

Residuals  108  171.8000  1.59074 


Interaction  plots  for  scenario:environment,  scenario:mode,  and  environment:mode. 


ANOVA  FOR  CHEMICAL  KILLS 


Analysis  of  Variance  Table 
Response:  MOE 


Terms  added  sequentially  (first  to  last) 


Df 

Sum  of  Sq 

Mean  Sq 

F  Value 

Seen 

2 

15811.47 

7905.73 

945.850 

Env 

1 

11427 . 01 

11427.01 

1367.140 

Mode 

1 

4002 . 07 

4002.07 

478 . 813 

Seen : Env 

2 

12043.47 

6021.73 

720.447 

Seen : Mode 

2 

1383.20 

691.60 

82.744 

Env : Mode 

1 

1026.68 

1026.68 

122.833 

Seen : Env : Mode 

2 

540.00 

270 . 00 

32.303 

Residuals 

108 

902.70 

8.36 

Pr  (F) 
0 . 000000e+000 
0 . 000000e+000 
0 . 000000e+000 
0 . 000000e+000 
0 . 000000e+000 
0 . 000000e+000 
1 . 008882e-011 


Interaction  plots  for  scenario:environment,  scenaricxmode,  and  environment: mode. 
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ANOVA  FOR  LERs 


Analysis  of  Variance  Table 
Response:  MOE 


Terms  added  sequentially  (first  to  last) 


Df 

Sum  of  Sq 

Mean  Sq 

F  Value 

Seen 

2 

5.75032 

2.87516 

14.90381 

Env 

1 

10.63265 

10 . 63265 

55.11590 

Mode 

1 

4.81601 

4.81601 

24.96450 

Seen : Env 

2 

13.11248 

6.55624 

33.98523 

Seen : Mode 

2 

3.39263 

1.69632 

8.79310 

Env : Mode 

1 

1.37388 

1.37388 

7.12171 

Seen : Env : Mode 

2 

1.30240 

0.65120 

3.37560 

Residuals 

108 

20.83476 

0.19291 

Pr  (F) 
0.00000192 
0.00000000 
0 . 00000226 
0.00000000 
0.00028975 
0 . 00878978 
0.03784347 


Interaction  plots  for  scenario:environment,  scenario:mode,  and  environment:mode. 
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ANOVA  for  Detection  Range  Blue 


Analysis  of  Variance  Table 
Response:  MOE 

Terms  added  sequentially  (first  to  last) 

Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 

Seen  2  95.34164  47.67082  6050.279  0.0000000 

Env  1  2.51797  2.51797  319.576  0.0000000 

Mode  1  4.11438  4.11438  522.188  0.0000000 

Seen : Env  2  0.19275  0.09638  12.232  0.0000163 

Seen : Mode  2  1.64627  0.82313  104.470  0.0000000 

Env : Mode  1  0.09369  0.09369  11.892  0.0008050 

Seen : Env : Mode  2  0.00861  0.00431  0.547  0.5805110 

Residuals  108  0.85094  0.00788 

Interaction  plots  for  scenarioienvironment,  scenario:mode,  and  environment: mode. 
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AISTOVA  for  Detection  Range  Red 

Analysis  of  Variance  Table 
Response:  MOE 

Terms  added  sequentially  (first  to  last) 

Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 

Seen  2  19.16607  9.58304  588.7363  0.00000000 

Env  1  5.15815  5.15815  316.8920  0.00000000 

Mode  1  12.66121  12.66121  777.8444  0.00000000 

Seen : Env  2  0.25595  0.12797  7.8622  0.00064909 

Seen : Mode  2  3.52387  1.76194  108.2451  0.00000000 

Env : Mode  1  0.15869  0.15869  9.7494  0.00230238 

Seen : Env : Mode  2  0.08810  0.04405  2.7063  0.07131454 

Residuals  108  1.75795  0.01628 

Interaction  plots  for  scenario:environment,  scenario:mode,  and  environment:mode. 
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ANOVA  for  Kill  Range  Blue 


Analysis  of  Variance  Table 
Response:  MOE 


Terms  added  sequentially  (first  to  last) 


Df 

Sum  of  Sq 

Mean  Sq 

F  Value 

Seen 

2 

14 . 17509 

7.087543 

111.0971 

Env 

1 

0.43093 

0.430931 

6.7548 

Mode 

1 

2 . 68719 

2.687186 

42.1216 

Seen : Env 

2 

0.31075 

0.155376 

2.4355 

Seen : Mode 

2 

0 . 62021 

0.310106 

4.8609 

Env : Mode 

1 

0 . 02739 

0 . 027389 

0.4293 

Seen: Env: Mode 

2 

0 . 18702 

0 . 093509 

1.4658 

Residuals 

108 

6 . 88996 

0 . 063796 

Pr  (F) 
0.0000000 
0.0106554 
0.0000000 
0.0923476 
0.0095194 
0.5137113 
0.2354604 


Interaction  plots  for  scenario:environment,  scenario:mode,  and  environment: mode. 


C\i 


Jlink 

Janus 
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Analysis  of  Variance  Table 
Response :  MOE 


Terms  added  sequentially  (first  to  last) 

Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 


Seen  2 
Env  1 
Mode  1 
Seen: Env  2 
Seen: Mode  2 
Env : Mode  1 
Seen : Env: Mode  2 
Residuals  108 


62.94847  31.47423 
0.32467  0.32467 

1.24480  1.24480 

0.33175  0.16587 

0.25323  0.12661 

0.30617  0.30617 

0.13178  0.06589 

2.38211  0.02206 


1426.979  0.00000000 
14.720  0.00021033 
56.437  0.00000000 
7.520  0.00087544 
5.740  0.00427329 
13.881  0.00031162 
2.987  0.05460434 


Interaction  plots  for  scenario:environment,  scenario:mode,  and  environment:mode. 


ANOVA  for  Rounds  Fired  Blue 


Analysis  of  Variance  Table 
Response:  MOE 


Terms  added  sequentially  (first  to  last) 


Df 

Sum  of  Sq 

Mean  Sq 

F  Value 

Seen 

2 

26900478 

13450239 

834.6258 

Env 

1 

1207813 

1207813 

74.9482 

Mode 

1 

192240 

192240 

11.9290 

Seen : Env 

2 

1821263 

910631 

56.5073 

Seen : Mode 

2 

151142 

75571 

4.6894 

Env : Mode 

1 

110231 

110231 

6.8402 

Seen : Env : Mode 

2 

183874 

91937 

5.7049 

Residuals 

108 

1740452 

16115 

Pr  (F) 

0 . 00000000 
0.00000000 
0.00079057 
0.00000000 
0.01114398 
0.01018746 
0.00441275 


Interaction  plots  for  scenario:environment,  scenario:mode,  and  environment: mode. 


100 


ANOVA  for  Rounds  Fired  Red 


Analysis  of  Variance  Table 
Response:  MOE 


Terms  added  sequentially  (first  to  last) 


Df 

Sum  of  Sq 

Mean  Sq 

F  Value 

Pr  (F) 

Seen 

2 

41699636 

20849818 

943.9545 

0.0000000 

Env 

1 

660677 

660677 

29.9115 

0.0000003 

Mode 

1 

65 

65 

0.0029 

0.9569931 

Seen : Env 

2 

1278612 

639306 

28.9439 

0.0000000 

Seen: Mode 

2 

180512 

90256 

4.0863 

0 . 0194671 

Env : Mode 

1 

30401 

30401 

1.3764 

0.2433019 

Seen : Env : Mode 

2 

367668 

183834 

8.3229 

0.0004348 

Residuals 

108 

2385475 

22088 

Interaction  plots  for  scenario:environment,  scenario:mode,  and  environment: mode. 
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fascam3.N.ja  Iascam3.swa.ia  fascam2.hl.ja  fascam2.swa.ja  (ascaml.hl.ja  (ascaml.swa.ja 


APPENDIX  F.  NORMAL  QUANTILE-QUANTILE  PLOTS 


FASCAM.Scen  #1,HL, Janus 


FASCAM,Scen  #1,HL,JLINK 


FASCAM.Scen  #2, SWA, Janus 


Quantiles  o<  Standard  Normal 


FASCAM.Scen  #3,SWA,Janus 

FASCAM.Scen  #3.SWA,JLINK 

5  . _ : 

s  5 

1  5 

1  s 

^-iT"  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  o(  Standard  Normal 

-1.5  -1.0  *0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal 
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Chem  Kills, Seen  #1, SWA, Janus 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal 


Chem  Kills, Seen  #1,HL, Janus 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal 


Chem  Kills, Seen  #2, SWA, Janus 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal 


Chem  Kills, Seen  #2, HL, Janus 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.S 

Quantiles  of  Standard  Normal 


Chem  Kills, Seen  #3, HL, Janus 


-1.5  -1.0  -0.5  0.0  0.S  1.0  1.5 

Quantiles  of  Standard  Normal 


chem3.hl.jt  chem3.swa.jl  chem2.hl.jl  chem2.swa.|1  cheml.KI.jt  cheml.swa.jl 


Chem  Kills,Scen  #1,SWA,JLINK 


Chem  Kills.Scen  #1,HL,JLINK 


Chem  Kills, Seen  #2,SWA,JLINK 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal 


Chem  Kills, Seen  #3,SWA,JLINK 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal 


Chem  Kills, Seen  #3,HL,JLINK 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quan files  of  Standard  Normal 
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I#r3  N  ja  Isr3.swa.ja  ler2.W.ja  Ier2.swa.ja  lerl.hl.ja  lerl.swa.ja 


LER.Scen  #1,SWA,Janus 


LER.Scen  #1,SWA,JLINK 


Quantiles  cA  Standard  Normal 


LER.Scen  #1  ,HL, Janus 


-1-5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal 


LER.Scen  #2,HL, Janus 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal 


LER.Scen  #3,SWA,Janus 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  c A  Standard  Normal 


LER.Scen  #3, HL, Janus 


*15  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal 


-1.5  -1.0  -0.5  0.0  0.6  1.0  1.5 

Quantiles  of  Standard  Normal 


LER.Scen  #1,HL,JLINK 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal 


LER.Scen  #2.SWA.JUNK 


•1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  c f  Standard  Normal 


LER.Scen  #2,HL,JLINK 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  (A  Standard  Normal 


LER.Scen  #3, SWA, JUNK 


-1-5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal 


LER.Scen  #3,HL,JLINK 


■1-5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal 
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d6tmg3.hLb.ja  detrng3.swa.bja  detrng2.N.b.}a  detmg2.swa.b.ja  detmgl.N.b.ja  detmgl  .swa.b.ja 


Det  Range, Seen  #1,SWA, Blue, Janus 


Det  Range, Seen  #1,SWA,Blue,JLINK 


Quantiles  of  Standard  Normal 


Quantiles  of  Standard  Normal 


Det  Range, Seen  #1  .HL.BIue.Janus  Det  Range.Scen  #1  ,HL,Blue,JLINK 


-1.5  -1.0  -O.S  0.0  0.S  1.0  1.S  *1.5  -1.0  *0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal  Quantiles  of  Standard  Normal 


Det  Range, Seen  #2, SWA, Blue, Janus  Det  Range, Seen  #2,SWA,Blue,JLINK 


Quanbtes  of  Standard  Normal  Quantiles  of  Standard  Normal 


Det  Range, Seen  #2,HL, Blue, Janus  Det  Range.Scen  #2,HL,Blue,JLINK 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5  -1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal  Quantiles  of  Standard  Normal 


Det  Range,Scen  #3, SWA, Blue, Janus  Det  Range, Seen  #3,SWA,Blue,JLINK 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5  -1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal  Quantiles  of  Standard  Normal 


Det  Range, Seen  #3,HL,Blue,Janus  Det  Range.Scen  #3,HL,Blue,JLINK 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5  -1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal  Quantiles  of  Standard  Normal 
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detrng3.hl.r.ja  detrng3.swa.r.]a  delmg2.hi.r.ja  detrng2.swa.r.|a  datrngl.hl.r.ja  detrngl.swa.r.ja 


Det  Range.Scen  #1,SWA, Red, Janus 


Det  Range, Seen  #1,SWA,Red,JLINK 


Quantiles  of  Standard  Normal  Quantiles  of  Standard  Normal 


Det  Range.Scen  #1  ,HL, Red, Janus  Det  Range.Scen  #1  ,HL,Red,JLINK 


*1.5  *1.0  -0.5  0.0  0.5  1.0  1.5  -1.5  -1.0  *0.5  0.0  0.5  1.0  1.5 

Quantiles  ol  Standard  Normal  Quantiles  of  Standard  Normal 


Det  Range.Scen  #2, SWA, Red, Janus  Det  Range.Scen  #2,SWA,Red,JLINK 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5  -1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal  Quantiles  of  Standard  Normal 


Det  Range.Scen  #2,HL,Red,Janus  Det  Range.Scen  #2,HL,Red,JLINK 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.S  -1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal  Quantiles  of  Standard  Normal 


Det  Range.Scen  #3, SWA, Red, Janus  Det  Range.Scen  #3, SWA, Red, JUNK 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5  -1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal  Quantiles  of  Standard  Normal 


Det  Range.Scen  #3,HL,RedpJanus 


Det  Range.Scen  #3, HL, Red, JUNK 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5  -1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Normal  Quantiles  of  Standard  Normal 
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ki«fng3.hl.b.ja  ktt!mg3.swa.b.ja  kittrng2.hl.b.ja  killmg2.swa_b.ja  killmgl  .hlJa.ja  killmg1.swa.bj  a 


Kill  Range, Seen  #1  ,SWA,Blue, Janus  Kill  Range, Seen  #1  , SWA, Blue, JUNK 


Quantiles  ot  Standard  Norms!  Quanbles  ot  Standard  Normal 


Kill  Range, Seen  #1  ,HL, Blue, Janus  Kill  Range, Seen  #1  ,HL, Blue, JUNK 


Quantiles  ot  Standard  Normai  Quantiles  ol  Standard  Normal 


Kill  Range, Seen  #2, SWA, Blue, Janus  Kill  Range, Seen  #2, SWA, Blue, JUNK 


Quanta  les  ot  Standard  Normal  Quantiles  ot  Standard  Normal 


Quantiles  of  Standard  Normal  Quantiles  of  Standard  Normal 


Kill  Range, Seen  #3, SWA, Blue, Janus  Kill  Range, Seen  #3, SWA, Blue, JUNK 


Quantiles  of  Standard  Normal  Quantiles  of  Standard  Normal 


Kill  Range, Seen  #3,HL,Blue,Janus  Kill  Range, Seen  #3, HL, Blue, JUNK 


Quantiles  of  Standard  Normal  Quantiles  of  Standard  Normal 
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Kill  Range, Seen  #1,SWA,Red, Janus 


Kill  Range, Seen  #1  .SWA.Red.JLINK 


— ■*' 

.svva.r.j 

t  3.6 

. 

- — 

I  « 

_ - r— -r" 

Quantiles  erf  Standard  Normal 

Quantiles  erf  Standard  Normal 

Kill  Range, Seen  #1  ,HL,Red,Janus 

Kill  Range, Seen  #1,HL,Red,JUNK 

^ - - 

CO 

- - 

f 

g 

- 

— - — "  * 

** 

-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  orf  Standard  Normal 

-1.5  -1.0  -0.5  0.0  0.5  1.0  1.S 

Quantiles  orf  Standard  Normal 

Kill  Range, Seen  #2, SWA, Red, Janus 

Kill  Range, Seen  #2,SWA,Red,JLINK 

i  o  ■ 

s 

E 

. . .  . . 

kil 

2.B 

- - - - *  ’*  1 

Quantiles  dl  Standard  Normal 


Quantiles  erf  Standard  Normal 


Kill  Range, Seen  #3, SWA, Red, Janus 


Kill  Range, Seen  #3,SWA, Red, JUNK 


Quantiles  o<  Standard  Norma! 


Quantiles  erf  Standard  Normal 
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rnds3.hl.b.ja  rnd*3.swa.b.ja  rndsZ.hl.b.ja  irris2.swa.b.)a  rndsl  ,hl.b.}a  mdsl.swa.b.ja 


Rnds  Fired, Seen  #1,SWA,Blue,  Janus 


Rnds  Fired, Seen  #1 , SWA, Blue, JUNK 


-1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

Quantiles  of  Standard  Norma) 

Rnds  Fired, Seen  #1,HL,Blue, Janus 


-1.0  -0.5  00  0.5  1  0  1.S 

Quantiles  of  Standard  Normal 
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APPENDIX  G.  KOLMOGOROV-SMIRNOV  GOODNESS  OF  FIT  RESULTS 


Highlighted  portions  are  those  MOPs  which  produce  p-values<0.05  (a=0.05),  and 
thus  reject  the  null  hypothesis  that  the  sample  comes  from  a  Normal  distribution. 
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APPENDIX  H.  /  -TEST  FOR  COMMON  VARIANCE  RESULTS 


Highlighted  portions  are  those  MOPs  which  produce  p-values<0.05  (oc=0.05),  and 
thus  reject  the  null  hypothesis  that  the  two  samples  share  a  common  variance. 


Scenario#l,SWA 


FASCAM 

Chem  Kills 

LER 

Janus 

JUnk 

Janus 

JLink 

Janus 

JLink 

.31 

.29 

.97 

Scenario  #1,HL 


■ 

FASCAM 

Chem  Kills 

LER 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

P-Value 

lil 

.24 

.10 

Scenario  #2,  SWA 


FASCAM 

Chem  Kills 

LER 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

P-Value 

.46 

.051 

.14 

Scenario  #2,  HL 


FASCAM 

Chem  Kills 

LER 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

P-Value 

W 

.15 

.11 

Scenario  #3,  SWA 


FASCAM 

Chem  Kills 

LER 

Mode 

Janus 

JLink 

Janus 

JLink 

Janus 

JLink 

P-Value 

ill: 

or 

:.60 

Scenario  #3,  HL 


FASCAM 

Chem  Kills 

LER 

Mode 

Janus 

JLink 

Janus 

JUnk 

Janus 

JLink 

P-Value 

MS. 

. 

;oo 

.23 

113 


Scenario#!,  SWA 
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APPENDIX  I.  t- TEST  RESULTS 


Highlighted  portions  are  those  MOPs  which  produce  p-values<0.0083, 

( 0Cnew  =  aold  /  6  =  0.0083 )  and  thus  reject  the  null  hypothesis  that  the  two  sample  means  are 
statistically  similar.  Also,  the  1  -cxnew  =  99.17  percent  confidence  interval  (Cl)  is  provided  to 
show  the  extent  to  which  the  sample  means  are  similar  (dissimilar).  All  MOPs  failing  to 
reject  the  null  hypothesis  (p-value>0.0083)  produce  CIs  which  include  zero. 
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Scenario  #1,  SWA 
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_  _ _  Scenario#!, SWA 

DetRngBlue  Det  Rng  Red  Kill  Rng  Blue  Kill  Rng  Red  Rnds  Fired  Blue  RndsRiedRed 

Mode  Janus  JLink  Janus  JLink  Janus  JLink  Janus  JLink  Janus  JLink  Janus  JLink 
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(-1.13,-71) 
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(-.54,. 03) 

(17.52,87.48) 

(-129.14,-41.26) 
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Scenario  #2,  SWA 
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APPENDIX  J.  WILCOXEN  TEST  RESULTS 


Highlighted  portions  those  MOPs  which  produce  p-values<0.0083,  ( aold  16  =  0.0083 ) 
and  thus  reject  the  null  hypothesis  that  the  two  sample  means  are  statistically  similar. 
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Scenario#!,  SWA 
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